Two Parts That Work, Breaking Each Other

Three copies of the same model sat on the machine in my office at the end of August — identical weights, converted to three different levels of precision. I switched on an accelerator that had proved itself on another model days earlier. Two of the three got substantially faster.

The third did not move. No error, no warning, no line in any log. It answered every question correctly at 26.8 tokens per second, exactly as before — as though the feature I had just enabled were not installed.

Ten days earlier the same class of failure had been loud instead: a new model, three times faster than the one it replaced, crashing my own infrastructure the moment it tried to save its work. Silent is worse.

A component that fails quietly costs more than one that fails loudly

The accelerator is speculative decoding: a small, cheap model guesses the next few words and the large, expensive one checks the whole batch in a single pass. Good guesses buy several words for the price of one; bad guesses buy nothing. On the model I had wrestled with the week before, it was finally worth 1.8 to 1.9 times — after I found my first measurement of it had been wrong three times over, which is its own story in A Negative Result Is Not a Conclusion.

On structured output, two of the three builds were unambiguous. The 4-bit build went from 36.4 tokens per second to 55–60 — between 1.5 and 1.64 times faster. The 8-bit build went from 21.6 to 32–37, a gain of 1.54 to 1.71 times. Output stayed byte-identical to the unaccelerated version, so this was pure speed with nothing traded away.

The 6-bit build measured between 0.85 and 1.04 times its baseline — no gain, sometimes slightly slower than doing nothing. Its underlying speed was perfectly normal at 26.8 tokens per second. Only the feature was dead.

The lesson for businesses: a component that throws an error gets fixed on Tuesday. One that quietly declines to do its job gets paid for, deployed and reported as working — until somebody measures it. No log was going to tell me.

The parts came from different factories

The explanation was provenance. The 4-bit and 8-bit builds came from one community conversion pipeline; the 6-bit came from a different one.

That sounds like a detail about file formats. It is not. The accelerator predicts from the exact internal numeric patterns a model produces while it thinks, and it had been trained against the patterns the first pipeline's builds produce. A different conversion recipe shifts them slightly. The guesses stop landing and the whole mechanism degrades into overhead.

Every part worked correctly on its own terms. Neither was defective; they had simply never been built to fit each other, and nothing in either could notice.

I rejected the 6-bit build for use with the accelerator and set a standing rule: builds come from the same pipeline the accelerator was trained against. That is a procurement rule as much as a technical one. If you are combining a model from one supplier with an add-on from another — a plug-in, an optimiser, a fine-tune — the question is not "does each of these work?" but "who tested the pair?"

Your own tooling is the thing least likely to survive a new model

The loud failure was the same principle from the other direction. I was adding a much faster model for coding work alongside the large production one: a mixture-of-experts design that uses only about 3 billion of its 35 billion parameters per word, and mixes two kinds of attention layer — efficient ones for most, standard for roughly one in four.

The speed was real. At the lighter precision it took 37.7 GB on disk and decoded at 66 tokens per second; the heavier version took 70.2 GB and ran 34. The incumbent managed about 20. Reading a long prompt ran at 2,302 tokens per second — roughly three times the model it replaced, on the same hardware, for work I do daily.

Then it crashed. Not the model — my caching layer, custom code written and tested against a model with a different internal shape. That code assumed a saved cache slot would always be either fully populated or entirely absent. The new architecture leaves a slot partially empty as a matter of course, and my code had never seen that. The result was a low-level type error at the moment of saving.

For a business, this is the cost nobody budgets: the model upgrade is a download; the integration work around it is the actual project.

Take the win you can verify, hold back the fix you cannot

What I did next matters more than the bug. I deployed both precisions and repointed my coding assistant at the faster one, because that gain was measured. I wrote the fix for the caching crash and verified it against five new test cases plus the existing suite of 86 tests, all passing.

And then I left disk caching switched off for that deployment anyway, until the fix could be properly reviewed and landed rather than pasted into a path sitting next to production. That discipline — proven is not the same as shipped — is the subject of The Speedup I Built and Refused to Ship.

Keeping those two decisions separate is the discipline. "This model is faster" and "my infrastructure is ready for it" are different claims with different evidence, and you can act on the first while the second is pending. What you should not do is let a patch verified by its own author at 11pm ride into a live system on the momentum of a good benchmark.

A process that is still running is not a process that is making progress

One more thing surfaced that day. The download of the heavier model — 70.2 GB of weights — stalled completely for about eight minutes. The transfer stayed alive at near-zero CPU throughout, looking exactly like healthy work; several of its parallel connections had silently died and nothing above noticed. The only check that caught it measured whether the bytes on disk were growing.

Liveness is not progress. If your monitoring confirms a job is running rather than that its output is growing, it will report that all is well for as long as you will wait. The same liveness gap hides inside prompt caches that quietly get slower as they fill — Five Seconds of Dead Air is what it looks like from the user's side.

The bottom line

Individually correct parts are not a working system. The most expensive failures I see are not broken components — they are well-behaved components never built to fit each other, degrading quietly while every dashboard stays green.

Measure the combination, not the parts. Nothing else will tell you.

For the engineers

Speculative decoding across three quantisations of one model. Draft-assisted decode on structured output: 4-bit 55–60 tok/s against a 36.4 baseline (1.5–1.64×); 8-bit 32–37 against 21.6 (1.54–1.71×); 6-bit 0.85–1.04× against a healthy 26.8 baseline. Outputs stayed byte-identical throughout.

Root cause was conversion provenance. The 4-bit and 8-bit weights came from one community conversion; the 6-bit from another. The draft head predicts from the base model's hidden-state distribution, and a different quantisation recipe perturbs that distribution enough to collapse the acceptance rate. No exception, no warning, no degraded-mode log line anywhere in the stack — only a throughput measurement separates a working draft head from a dead one. Verdict: standardise on the pipeline the draft head was trained against; treat cross-pipeline pairings as unverified until benchmarked.

The earlier failure was a serialisation assumption. A two-tier RAM-plus-disk prompt cache assumed each per-layer slot was either fully populated or absent when saving state. The new model interleaves linear-attention and full-attention layers, roughly one full in four, which legitimately produces a partially-empty slot; saving it raised a low-level type-cast error. Fix verified against 5 targeted cases plus 86 existing tests, all green — then deliberately kept out of the production path, disk caching disabled for that deployment, pending review.

If you are pairing an AI model with an accelerator, a plug-in or a caching layer that came from somewhere else — or upgrading to a new architecture while running integration code written for the last one — that pairing is what I would spend a consulting day on, for your business in Dubai, the UAE, or wherever the stack runs. I will benchmark the combination rather than the parts, find the features that are silently doing nothing, and tell you which of your infrastructure survives the upgrade and which quietly does not. Book a consulting day or send me an inquiry listing the pieces you have bolted together.