A Negative Result Is Not a Conclusion

There is a well-known technique for making a language model answer faster, with published numbers behind it: roughly 2.4 times quicker, on hardware identical to mine. I switched it on, ran it across three prompt sizes, and measured about 28 tokens per second — the same as with it switched off, inside the noise.

That is a clean negative result, and the professional move is to write it up in one line — evaluated, no measurable benefit — and get on with the week.

I didn't file that line. The gap between "2.4 times" and "nothing at all" was too wide to shrug at. By the end of the day the same feature was running 1.8 to 1.9 times faster on the workloads it suits, with nothing about it changed. My measurement had been wrong three times over, for three unrelated reasons.

My benchmark measured my filler text, not the feature

The technique works by guessing. A small, cheap model drafts the next few words; the large, expensive one checks the whole guess in a single pass. Correct guesses are nearly free; wrong ones cost the drafting time for nothing. Its entire value depends on how predictable your text is.

My benchmark padded its prompts with repetitive filler, on the assumption that padding is neutral. It was the worst possible case: the draft model was right on only 32 to 47 percent of its attempts, under the point where the technique pays for itself. Fed natural prose and code, the same setup was right 63 to 73 percent of the time and delivered that 1.8 to 1.9 times speedup.

The lesson for businesses: test data is a decision input, not a formality. Anything whose performance depends on the shape of your data has to be measured on your real documents. Synthetic filler buys a confident number about a system nobody will ever operate. The same inversion showed up when the top-ranked document model finished last on my real pages — The Top-Ranked Model Was the Worst One for My Documents.

A safety default held the whole thing at its own floor

The second cause was a default. The software adjusts how many words it drafts at a time, based on how often its guesses are accepted. It starts at a floor of three and grows past that only once acceptance passes 85 percent. On my workload, 85 percent is unreachable — so it sat at three forever, in its most cautious setting, delivering no real speedup.

Nothing was broken. A rule written for someone else's traffic simply never fired on mine.

For a business: "adaptive" and "automatic" describe a policy someone else wrote, with a trigger nobody has checked against your numbers. If it never fires on your traffic, the feature is off while every dashboard insists it is on.

The bug that mattered was not in the feature I was testing

The third cause was worth the whole day. The checking step — where the large model verifies a batch of guesses — was taking 2.7 seconds, and varied from request to request. It had no business varying.

Running the identical request two ways, through the library directly and through the server that handles live traffic, gave two different timings. The server was never reserving its memory properly, so under load the model's own weights were pushed out and pulled back in on every verification round.

Once fixed, the checking step went from 2.7 seconds to 0.08, and generation from 1.7 tokens per second to 38.8.

The part that should worry any operator: that misconfiguration had nothing to do with the experiment. It sat in the serving path of every model server on that machine, taxing real traffic daily — and nobody would have caught it by watching production, because production worked. It merely worked slower, with no faster version to compare against. A failed experiment is what found it.

I proved it worked and then left it switched off

I still did not enable the feature for the everyday assistant role, even at 1.8 to 1.9 times.

That role's traffic is short tool-calling turns: a few words, stop, run something, a few more words. On turns that shape, the drafting overhead costs more than the guessing saves. A verified speedup on the wrong traffic is a regression with a good reputation — the same arithmetic that made me leave another proven speedup switched off in The Speedup I Built and Refused to Ship.

What I kept was the memory fix, applied to every production server that day. A day later, on a different model, the same feature behaved differently again.

"Does it work" and "does it help us" are different questions, and vendors answer only the first. And be clear about limits: these are my figures, from one machine and one set of workloads. The shape transfers; the values will not.

Nobody had fixed it because nobody runs your workload

A few days later, after a run of similar finds in widely used open-source infrastructure, I put the uncomfortable question to myself: if these are such obvious wins, why hadn't the people who maintain the software already made them?

The honest answer comes in layers, and none of them is negligence. The heavy computation underneath, the part everybody uses, I ran exactly as shipped; nothing there needed fixing. The layer where the worst defect lived is a community-maintained project supporting around fifty model families, and the feature that misbehaved was weeks old. It was built and tested at a scale of tens of stored entries, where it is correct and fast, and grows steadily more expensive at a thousand. Nobody maintaining it had spent two days pushing hundred-thousand-token sessions into a terabyte-sized cache. I had.

That is the classic shape: invisible at the scale the author tested, dominant at the scale a heavy user reaches.

The lesson for businesses: the tools you depend on are tuned for the scale their maintainers could realistically test, which is almost always smaller than yours at full load. That is not a scandal, it is how shared software gets built — and closing the gap is what your own measurement is for. A tool's reputation is not a measurement of your traffic.

The bottom line

A negative result tells you what happened in your test. It does not tell you what would have happened in a correct one. Before shelving a tool or a vendor on one flat number, spend an hour interrogating the number itself — the test data, and the defaults underneath it. Mine was wrong three times over, and the third mistake was costing me money on work that had nothing to do with the experiment.

For the engineers

The technique is speculative decoding: a draft model proposes k tokens, the target verifies them in one forward pass, and accepted tokens emerge at roughly verification cost. Throughput tracks acceptance rate, which tracks how predictable the text is.

Bug one was the harness. Repetitive filler put per-draft acceptance at 32–47%, below the break-even where drafting overhead pays for itself; three context sizes all came back at ~28 tok/s with the feature on or off, within ±15% noise. On natural prose and code the same configuration accepted 63–73% and returned 1.8–1.9× end to end.

Bug two was the adaptive block-size controller: it starts at a floor of three draft tokens and only grows when acceptance reaches 85% — unreachable here — so block size never left the floor and speedup sat near 1×.

Bug three mattered. Verification latency was 2.7 s per step and variable, but only through the serving path; the same request via a direct library call behaved. That process never set a wired-memory limit, so under pressure the weight buffers were evictable and re-faulted every verification round. With the limit set: verify 2.7 s → 0.08 s, decode 1.7 → 38.8 tok/s.

Check that your inference servers set a wired limit explicitly. The default is unset, it raises no error, and it never shows up on a dashboard. The follow-on discovery — that the same accelerator silently does nothing across mismatched conversion pipelines — is in Two Parts That Work, Breaking Each Other.

If your team benchmarked a promising model or vendor once, got a flat result and quietly moved on — or if nobody has ever audited the defaults underneath your AI stack — one consulting day usually settles it, whether you're in Dubai or running this across your business remotely. I will re-run the measurements on your real traffic, separate "this doesn't work" from "we measured it wrong", and leave you the settings quietly costing you throughput. Book a consulting day or send me an inquiry and send me the benchmark you shelved.