The 6.3-Point Gap That Wasn't: 132 Runs on a Cheaper Model

I run the same open-weight model at two precisions on my own hardware. One is the full-precision original. The other is a quantized copy — the same model, compressed, so each number inside it is stored with less detail. The compressed one generates text about twice as fast and costs roughly half as much to keep running.

So the only question that mattered was: is the expensive one actually smarter? Not faster. Not more comfortable. Smarter.

I had an opinion. I'd been switching between them for weeks and the full-precision version felt sharper — it seemed to reason a little further, catch a little more. That feeling was worth exactly nothing, and in August I finally admitted it and built a proper benchmark instead. A hundred and thirty-two runs later I had an answer, and it was not the one I'd been carrying around.

Spot-checks are a random number generator with an opinion

Here's how almost every model comparison in business actually gets made. Someone runs six or eight prompts through both options, reads the answers, and forms a view. Usually one model flubs something memorable. That flub becomes the verdict.

The problem is that language models are non-deterministic. Ask the same model the same question three times and you get three slightly different answers, sometimes one right and two wrong. A handful of side-by-side prompts is a sample small enough that ordinary variation looks exactly like a quality difference. You are not measuring the models. You are measuring which model happened to have a bad day in front of you.

This is not a subtle effect. It is the dominant effect at small sample sizes. It is how companies end up paying double for a "smarter" model that is not smarter, or committing a workload to a cheap model that will quietly fail in a way nobody tested for — and the quiet failure usually comes from the shape of the call rather than the intelligence behind it, as the local model that passed chat and failed my pipeline showed.

If you are going to make a real decision — pay double, migrate a pipeline, sign a year of capacity — you need a benchmark that can tell a real gap from noise. That means enough repetitions to have a sample, a grader that doesn't have a favorite, and a statistical test at the end to check whether the number you're looking at means anything at all.

What a real harness actually looks like

Toy prompts don't work. "Write me a haiku about teamwork" tells you nothing about whether a model can operate inside your business.

So I built a purpose-made test repository: a small codebase with a genuine bug in it and several planted decoys — plausible-looking wrong answers sitting near the real one, so a model that pattern-matches instead of investigating gets caught. Twenty-two tasks in total, spanning three things I actually care about:

• Agentic tool use. Can it navigate a real repository, read the right files, and find the actual defect rather than the nearest convincing one?
• Open-ended discussion. Given an ambiguous question, does it reason usefully or produce confident mush?
• Exact-format instruction following. When I say "reply with only this string, no commentary," does it comply, character for character?

Each task ran three times per model. Twenty-two tasks, three trials, two precisions: 132 runs.

Three trials per task is not statistical luxury. It is the minimum that lets you tell a model's actual behavior from a bad roll of the dice. A single spot-check records whichever roll it happened to catch and calls it the truth.

The rules that make the answer trustworthy

The harness is the easy part. The discipline around it is what makes the number worth believing.

Graders validated before spending any inference time. For every task with a right answer I wrote a deterministic grader — code, not judgement — then tested the grader against a known-good answer and a known-bad one before letting it near real results. A grader that marks everything correct is worse than no grader, because it produces a number you'll trust.

Blind judging for the open-ended items. The tasks with no mechanical right answer were scored with the model labels sealed until scoring was finished. I knew which precision I expected to win. That expectation would have leaked into every borderline call I made, and borderline calls were most of them.

Both models running concurrently on independent backends. Not one after the other. Running sequentially means the second model gets a warm machine, or a cold one, or one that's busy with something else — and you have no way to separate that from capability. Same conditions, same clock, same load.

Significance testing decided in advance, so I couldn't go looking afterwards for the test that gave me the answer I wanted.

None of this is exotic. It's the ordinary hygiene of measuring anything. It's just that almost nobody applies it to model selection, which is one of the more expensive decisions on the table.

The early lead that evaporated

Partway through, the expensive model was ahead on something concrete: six tool errors for the cheap model against zero for the pricey one. Six to nothing. That's the kind of gap you screenshot and act on.

It was an artifact. The two models hadn't completed the same number of operations yet — one had simply done more work and therefore had more chances to slip. Once both had run the same volume, the tally was 11 errors out of 113 for one and 9 out of 105 for the other. Both under ten percent, about a point apart. Identical, for practical purposes.

That collapse is the most useful thing that happened all day. A six-to-nothing gap felt like a finding. It was a counting error dressed up as insight, and the only reason I caught it is that I kept running instead of stopping when I had a result I liked.

The headline number failed three separate tests

Final scores: the cheaper quantized model at 79.4%, the full-precision one at 85.7%. A 6.3-point gap in favor of the expensive option.

That looks decisive. Six points is the kind of margin that justifies doubling a cost line.

It failed three separate significance tests, with p-values of 0.35, 0.28 and 0.29. A p-value is the odds of seeing a gap this big by pure chance when the two things are actually equivalent; anything above roughly 0.05 means you cannot rule out coincidence. At 0.29 you are nowhere near it. With 22 tasks, a 6.3-point spread is exactly the size of gap that random variation produces on its own.

So the honest conclusion was: no measurable quality difference, at this sample size, on this workload. The expensive model may well be better. My benchmark could not show it, and neither could anyone's six-prompt eyeball test.

The defect that did survive — and nobody would have predicted it

Here's the part I didn't expect.

The failure I was hunting — the quantized model being dumber, reasoning less well, walking into the planted decoys — never appeared. What appeared instead was narrow, strange, and completely real: three times across the full run, the cheap model reasoned correctly and then garbled the literal characters of its answer. A transposed character inside an identifier. A dropped character in a regular expression. One task it had genuinely solved, answered with a stray symbol instead of a plain statement that it had worked.

Three of those in the cheaper model. Zero in the full-precision one.

Compression damage doesn't show up as stupidity. It shows up as a smudged photocopy: the reasoning is entirely intact, and then the ink slips on a couple of letters. Think of a flawless dictation in which the single number in it comes out with two digits swapped. The thinking was right. The transcription wasn't.

That's a completely different risk profile, and it points at completely different use cases. For drafting, summarizing, classifying, discussing — cases where a human reads the output and a swapped character is obvious and harmless — the cheap model is fine, twice as fast, and half the cost. For anything where the exact string is the product — code, identifiers, regular expressions, structured output going straight into another system with no human in between — three slips in 132 runs is a real defect that an aggregate score of 79.4% will never tell you about.

The lesson worth stealing

Define the failure mode you actually care about before you benchmark, because the summary score will not show it to you.

Both of my numbers were true and neither was useful on its own. The 6.3-point gap was noise that looked like signal. The three character-level slips were signal small enough to vanish inside any headline percentage — and they were the entire finding.

So before you compare models, write down the specific way a wrong answer hurts your business. Is it a bad recommendation a human will catch? Or a malformed identifier that silently corrupts whatever it is written into? Those demand different tests, and the second one is invisible to the first.

I've written before about where local open models sit against the cloud frontier; this is the layer underneath that argument — how you would actually know. The companion question when a benchmark comes back with a real failure is what the model does when there is nothing to find at all, which is where a vision model wrote confident captions for blank images. For companies in Dubai and across the UAE weighing a cheaper model, an in-region option, or a local deployment, the same split decides it. A model that swaps two characters is harmless drafting a reply a person will read and expensive writing an identifier into a system that acts on it before anyone looks. The options on the table are close enough on quality that the choice turns on which of those two jobs you are buying it for, not on a leaderboard — and a vendor demo is a spot-check with a sales incentive attached.

I ran 132 blind runs to avoid paying double for a difference that didn't exist. Building a real evaluation harness for one decision like that took an afternoon — genuinely the scope of it. So if you're about to pay double, migrate a pipeline, or sign a year of capacity on the strength of a demo, that's a day on-site to build a test set from your own cases and tell you whether the gap you're paying for exists. Book a consulting day or send me an inquiry first if you'd rather talk before booking.