The Model That Couldn't Say \"I Found Nothing\" — It Wrote a Caption Instead

A few months ago I found a hole in the internal search behind one of my businesses: it could not see pictures. The extraction pipeline underneath it only handled formats it could pull text out of directly, so every image in years of accumulated material had been passed over without a word of complaint. Not failed. Skipped. Tens of thousands of files sitting in storage with no searchable text attached to them at all.

So I built the missing piece: an OCR pass that reads the text out of images and stores it alongside everything else. Straightforward work. What came out of it was the most useful reminder I have had all year about how AI tools fail, and it had nothing to do with accuracy on the happy path.

The benchmark took an afternoon and settled the default

Two candidates. First, a vision-language model running locally on my own hardware — a model that looks at an image and writes out what it sees. Its transcriptions were excellent: messy scans, handwriting, odd layouts, all handled. Local was the right first look for exactly the reasons I set out in Cloud vs. Local Models in 2026: the data never has to leave the building.

It took 31.2 seconds per image.

Second candidate: the text-recognition engine already built into the operating system. No model to download, no GPU time, no tokens. It ran at 0.135 seconds per image, at a quality I judged comparable on the same test images.

That is a 245x difference. On one archive of roughly 31,000 images, the smart model would have needed about eleven days of continuous processing. The built-in one finished the same set in about seventy minutes.

There is a lesson sitting in that gap, and it is the same one I keep writing about dumb components beating clever ones. But that was not the finding. The finding came from the test set.

Build a test set that includes nothing

When I benchmark anything I build a small stratified set — a deliberate spread of the cases I expect, not a random handful. Forty images: dense text, sparse text, handwriting, low-quality scans, and five images that contained no text whatsoever. Photographs. Nothing written anywhere in them.

I included the zero-text images to settle a design question I already had. If the fast recognizer comes back empty, should I hand the image to the slower, smarter model as a second opinion? That is the natural instinct. Cheap pass first, expensive pass on the hard cases.

Four of those five zero-text images came back from the vision model with a confident, fluent caption describing the photograph.

Not an error. Not an empty string. A paragraph of clean prose that would have been written into the text field of my search index, in exactly the same column as real transcriptions, with nothing to distinguish it from words a human being had actually put on a page.

This is a design problem, not a prompt problem

My first reaction was the wrong one, and it is the reaction almost everyone has: I will just tell it not to do that.

I was already using a structured output schema — a strict format the model has to fill in rather than free prose. I made the instruction explicit: transcribe text only, never describe the image, return empty if there is no text. The captions kept coming.

This is the part worth internalizing, because it survives contact with better models and cleverer prompts. A language model asked to produce output will produce output. "Nothing" is an answer it is structurally reluctant to give, and no amount of polite instruction reliably overrides that. You are asking a machine built to continue a pattern to stop mid-sentence and admit the page is blank.

The only defense that works is not letting the model into the room. You do not prompt your way out of hallucination. You design the pipeline so the model is never in a position where guessing is its best available move.

The trigger I shipped instead

So I inverted the escalation rule. The slow model does not run when the fast pass finds nothing. It runs when the fast pass finds something and reports low confidence per character — a smudged scan, a bad angle, a faded page.

That case is structurally immune to the failure I had just found. There is demonstrably text in the image; the fast engine saw it and struggled with it. Asking a better reader to have another go at genuinely present text cannot produce an invented caption, because the model is not being handed a blank wall and asked what is written on it.

Same two components, same two models, completely different risk profile — purely from where the boundary between them sits.

The cost side of that choice was measurable too. The naive rule — escalate whenever the fast engine found little or no text — would have fired on 42 to 60% of all images depending on the archive. The confidence rule fired on roughly 5 to 25%. So the design that removes the fabrication risk is also, by a wide margin, the cheaper one. That is not a coincidence. Rules that trigger on absence trigger constantly, because absence is common.

One more number, because it is a useful humility check: my first small-sample estimate of the escalation rate was 11%. On the full archive it was 20%. Small samples tell you whether something works. They do not tell you what it will cost.

Two traps sitting right next to the first one

Two more failure modes turned up in the same stretch of work, and both are invisible until you go looking.

A context window that was silently too small. The smaller model I tested had been configured with a working memory too small for the large images I was feeding it. It did not error. It did not warn. It truncated the input and answered anyway — from a fraction of the page, blind, with total confidence. Same disease as the fabricated captions, different cause: a system that cannot say "I could not see all of this" will simply invent the part it missed.

Runaway generation on one bad image. During an overnight backfill, a single image sent the model into a loop, generating text it had no business generating, and stalled the entire job for over an hour before I capped it. The fix took two minutes: a hard ceiling on output length. Any AI process that runs unattended needs one, because "it will stop when it is done" is an assumption, not a guarantee.

The verdict I got wrong

Here is the correction I owe the slow model. That 31.2 seconds per image was measured cold — first run, model loading from disk, nothing warmed up. Benchmarked properly, warm, it ran between 1.3 and 8.7 seconds depending on which model I used. Still an order of magnitude behind the built-in pass, but a different universe from eleven days.

That does not change the default. It does turn a firm no into a viable selective fallback, which is exactly what the low-confidence path needed. Benchmark warm, or you will make architecture decisions on a startup cost you only pay once.

The finished system now has just over 100,000 images across the archives it covers carrying searchable text, most of it produced by the boring engine that came free with the machine.

What to ask before you buy any extraction tool

If your company is evaluating AI for reading scanned or photographed material — anything that ends up in a search index or a record someone later trusts — the demo will run on the happy path. Every demo does. Ask for the other one.

1. Show me what it returns when there is genuinely nothing to find. Hand it a blank page, a photo, a corrupted file. The right answer is an empty result or an honest error, not fluent prose.
2. Show me how confidence is reported per field, and what the system does with a low score. If everything comes back as clean text with no confidence attached, you cannot build a review queue on it.
3. Show me the cap. What stops one bad input from consuming an unbounded amount of time and money at three in the morning?

Fabricated output in a search index is the quietest kind of damage there is. Nothing breaks, no alert fires; the records simply become subtly untrue, and you find out months later when someone acts on one. That risk bites harder here than most places, because companies across Dubai and the UAE are digitizing years of accumulated back-office material at speed, and an extraction layer that invents content poisons the well for every process downstream that trusts it.

The other half of doing this evaluation properly is making sure the quality gap you're designing around is real in the first place — 132 blind runs to settle one model question is the methodology piece that pairs with this one.

If you are about to point an AI tool at an archive of stored files and want the evaluation done properly before it runs on a hundred thousand files, that is a consulting day. Book a consulting day or send me an inquiry first if you'd rather talk before booking.