Buying AI: A Field Guide for Non-Technical Leaders

Every week another AI tool lands in your inbox promising to transform your business. Most of the people pitching them don't understand your business, and most of the people evaluating them on your side don't understand the technology. That gap is where budgets go to die.

I've sat on both sides of this table — building AI systems and reviewing vendors' claims. Here's the field guide I wish every non-technical leader had before signing anything.

First, sort the claims into three buckets

Almost every AI product promise falls into one of three categories, and each deserves a different level of skepticism:

Bucket 1: Solved problems. Transcription, translation, summarization, first-draft generation, code assistance. These work, reliably, today. If a vendor is selling these, the question isn't "does it work?" but "is it cheaper or better than the three other tools selling the same thing?"

Bucket 2: Possible but fragile. Anything where the AI reads your documents and answers questions (RAG), anything autonomous that touches customers, anything that must be right 100% of the time — legal clauses, medical dosing, financial figures. These can work, but they require real engineering around the model: evaluation, guardrails, human review. A vendor demoing this category is showing you their lucky day. Ask for the failure rate.

Bucket 3: Marketing fiction. "Self-improving AI agents that learn your business." "99% accuracy on any document." "Fully autonomous decision-making." If a claim has no failure mode attached, the vendor either hasn't tested it or is hiding something. Walk away.

The five questions that cut through everything

1. What happens when it's wrong? Not if — when. Every AI system is wrong sometimes. A serious vendor answers this instantly and specifically: what the error looks like, how often it happens, what catches it. Evasiveness here is the single best predictor of a bad purchase.

2. What data does it need, and what happens to it? You'd be amazed how many "enterprise-ready" tools ship your data to third-party servers by default. Where is it processed, is it used for training, can it be deleted? For a UAE business, add: does processing stay in a jurisdiction you can defend to your clients?

3. What does it cost at real volume? Demo pricing is per-seat trickle pricing. Ask for the cost at your actual monthly volume, including the API bills their architecture will generate on your account, if any. Then ask what the cost looks like if usage succeeds and triples — usage-based billing has a way of quietly routing your traffic through the most expensive model (your cheap path runs on the expensive model).

4. What happens if this vendor disappears? AI vendors fail constantly. Is there an export path for your data and configurations? Can the workflow run without them? Lock-in with a young company is a bigger risk than any technical limitation.

5. Who inside my company owns this? AI tools die not from bad technology but from missing ownership. If you can't name the person whose KPI improves when this works, don't buy it yet.

The evaluation ritual that actually works

Skip the demo. Instead, run a two-week pilot on your data, with your worst cases — not the vendor's cherry-picked samples. Specifically:

• Give it the messiest, most ambiguous 10% of your real workload, not the clean 90% every demo uses
• Define success numerically before you start ("handles 80% of invoices without human correction") — and remember that a test is not a load
• Count the human minutes the system saves after correction effort — this is the only number that matters
• Check what it costs when something breaks: support quality, error recovery, your team's time

If a vendor won't support a structured pilot like this, that tells you everything.

Where consultants fit in this picture

A good AI consultant isn't there to pick tools for you — that list changes monthly anyway. The value is in the questions above: stress-testing claims against your actual workload, spotting the bucket-3 fiction before the contract is signed, and estimating the real engineering cost hidden behind "just plug it in."

The math is simple: an AI vendor sells you their product (build vs. buy vs. blend). An independent advisor — someone with no product to sell you — is the only person at the table whose incentive is your outcome. That's exactly why I structure my consulting engagements the way I do: one focused day, your real problems, no agenda beyond making the right call obvious.

If you're currently weighing an AI purchase or vendor contract, a single hour on a call is usually enough to know which bucket you're dealing with. Or send an inquiry describing what you're evaluating.