The Top-Ranked Model Was the Worst One for My Documents

I wanted document recognition running on my own hardware — invoices, letters, forms. Mostly Arabic, usually with a line of French or English running through them and a table of numbers at the bottom. Ordinary Dubai paperwork.

So I started where everyone starts: the model at the top of the public document-OCR leaderboard. About a gigabyte on disk, open weights, ranked first. On a clean English test page it read 158 words with a single error, in under two seconds. Impressive, for something that small.

Then I gave it a page of Arabic and French with a numeric table. It made 28 word errors out of the same 158 words — not stylistic slips, real letter-level mistakes — and got 16 of 22 table cells right. It was the worst of the four Arabic-capable local models I tested that day. Models ranked well below it on the general leaderboard beat it outright.

A leaderboard ranks the average, and your documents are not the average

A public benchmark score is one number standing in for performance across a broad spread of document types, layouts and languages. A model can lead that average and still be weak on one script, because that script is a small slice of the test set. Nobody is hiding anything. The number is honest — it just is not answering the question I was asking.

My question was not "which model is best at documents". It was "which model is best at these documents", and the answer inverted the ranking completely. The same day, a separate eval on my assistant models found its failures concentrated in exactly the places a leaderboard never tests — Thirty Tasks, Three Models, One Decision.

A leaderboard tells you how models compare on someone else's test set. If your decision hinges on one category — a language, a document format, a customer segment, a regulated form — the leaderboard-topper is a reasonable place to start looking and a poor place to stop.

The model that won cost ten times the disk and six times the wait

The candidate that actually handled my pages was a mid-sized open vision-language model: 8 billion parameters, Apache-licensed, about 11 GB on disk. On the same Arabic-and-French page with the numeric table it made 1 word error out of 158 and got 21 of 22 cells right.

It takes 12.6 seconds per page. That is roughly ten times the disk of the leaderboard leader and at least six times the wait.

It is the right trade almost every time. Twelve seconds is a coffee-sip. Twenty-eight wrong words in a customer's document is a wrong name on a contract, a wrong amount on an invoice, a wrong number carried silently into a spreadsheet — and unlike a slow page, nobody sees it happen.

The lesson for businesses: price the error, not the second. Latency is felt by one person once. A recognition error is felt downstream by everyone who trusts the extracted data, for as long as it survives uncorrected.

The fastest option was free — and it reversed my numbers

The best speed result of the day did not come from a model at all. The recognition already built into the machine's operating system read a clean English page in 0.35 seconds with zero errors, at no cost and with nothing to load. It read Arabic correctly too.

Then I looked at the mixed lines. Numbers and Latin characters embedded inside Arabic sentences came back in the wrong reading order — "11%" returned as "%11".

Read that again as a business problem rather than a typography one. The text is not garbled. It is not flagged. It looks fine to anyone reviewing it quickly, and the digits are all present. A percentage, a reference number or an amount has simply been reassembled backwards inside an otherwise perfect sentence. That is a separate bug needing a separate fix, and exactly the class of failure that survives a demo, survives a pilot, and surfaces later in a reconciliation nobody can explain. It is the sibling of the silent errors I found when a local model passed every chat test and failed the pipeline — The Local Model Passed the Chat Test.

In this region, mixed Arabic-and-Latin text is not an edge case, it is Tuesday. Any system you buy for document processing here has to be tested on real mixed-script pages with numbers in them, by someone who reads both — not on the vendor's sample invoice.

Time nothing while the machine is busy

One embarrassment worth recording. Early on, my timings were wild — until I realised I had left a large model download running in the background. It was slowing the model I was timing by up to ten times.

I paused the downloads and re-ran everything from the start. Different numbers.

The lesson for businesses: the measurements you collect during an evaluation can be wrong for reasons that have nothing to do with what you are evaluating. If a pilot result surprises you, ask what else was running on that machine, that network or that shift before you draw a conclusion from it.

I stopped looking for a winner and built three tiers

The honest reading of the day is that no single model won. Each candidate was best at something and unusable at something else, and there was one more data point: my large production chat model, at 320 billion parameters, is far too slow for bulk work — and was the only candidate in the whole test that never once reversed an embedded number.

So the setup I run is three tiers, not one choice. The instant built-in recognition handles plain, fast text where speed is the point. The 8-billion-parameter vision-language model handles documents and anything Arabic, on its own service, at 12.6 seconds a page. The big model sits behind both as a deliberate, slow fallback for the pages that have to be right — the ones where the number is the whole reason the document exists.

For a business: "which model should we use" is usually the wrong question. The right one is which document goes to which tier, and what it costs you when a page is routed to the cheap one and it is wrong.

The bottom line

A leaderboard ranks average performance on a test set that is not yours. On my Arabic pages the ranking did not just shift — it inverted, and the top-ranked model finished last of four. Testing on your own real documents costs an afternoon and answers a question no public benchmark can. Do it before you sign anything.

For the engineers

The test set was two pages: a clean English page and a mixed Arabic-and-French page with a 22-cell numeric table, both 158 words, scored on word errors and cells recovered. Four Arabic-capable local candidates, identical inputs.

The ~1 GB leaderboard leader: 1 word error on English in under 2 seconds; 28 word errors and 16 of 22 cells on the mixed page, with real letter-level substitutions rather than diacritic noise. The 8B Apache-licensed vision-language model: 1 word error, 21 of 22 cells, 12.6 seconds per page, ~11 GB resident. The OS-native recogniser: 0.35 seconds, zero errors on English, correct Arabic glyphs — but bidirectional reordering failures on embedded Latin and numerals, "11%" emitted as "%11". That is a bidi resolution defect in the text-assembly stage, not a recognition defect: the characters are right and the sequence is wrong, so the fix is a post-pass over the returned runs, not a better model. The 320B chat model, run as a slow fallback, was the only one that never reordered an embedded number.

Measurement hygiene mattered: concurrent large model downloads inflated per-page latency by up to 10x, which is more than enough to distort a comparison. Pause every background transfer before you time anything, and re-run the whole matrix rather than patching in outliers.

If your business runs documents through an AI system — invoices, contracts, forms, anything with Arabic and numbers on the same line — and nobody has checked it against pages you already know the right answers for, that is a consulting day ending in a routing plan, wherever you operate: Dubai, the wider Gulf, or beyond. I will build a test set out of your own real documents, run the candidates through it identically, and leave you a scorecard and a routing plan: which pages go to the fast tier, which have to go to the careful one. Book a consulting day or send me an inquiry with two pages you already know the right answers to.