The Speedup I Built and Refused to Ship

Two switches on the machine in my office sit in the off position. I wrote both, tested both, finished both on the same day. Both work.

One makes a specific, expensive job run more than twice as fast. On an identical cold prompt of 100,917 tokens of real text, the path I run in production took 648.5 seconds. The new path took 313.5. That is 155.6 tokens per second against 321.9 — a 2.07 times end-to-end speedup, with a maximum numerical difference against the old method of 0.0000.

I did not turn it on. That decision is the whole post, because it is the one I see most teams get backwards: a green benchmark is read as a shipping authorisation, and the day the number looks good is the day it goes live.

A measurement is only as good as its control

The speedup was not a lucky run. It widened exactly the way the theory said it should: 1.34 times faster at 32,000 tokens of context, 2.44 times at 65,000, 3.31 times at 98,000 — where one chunk of work dropped from 23.4 seconds to 7.1. Memory stayed flat, with no swap.

The part I trust most is the part where nothing happened. Below a threshold of 24,000 tokens, the two paths were the same speed to the byte. That is a built-in control: it proves the change is inert on small work rather than trading one size of job against another.

The lesson for businesses: a benchmark that only shows improvement is weaker evidence than one that also shows no change where no change was expected. If a vendor's numbers are all wins, ask what their test did in the cases the feature should not touch.

A speedup only pays on the traffic you actually have

Here is why the switch stayed off, and it has nothing to do with doubt about the measurement.

That change helps exactly one situation: picking up a long conversation cold. It does nothing for short, latency-sensitive requests — most of what that machine handles hour to hour — and nothing for continuing turns inside a warm session. The 2.07 times is real, and it applies to a slice of my day I could count on one hand.

Multiply every claimed speedup by the share of your work it actually touches. A three-times gain on eight percent of your volume is worth less than a five percent gain on all of it, and vendors quote the first number because it is bigger. Ask which of your jobs the improvement applies to, then ask how many of those you run. This is the same trap as a transcription model that is 2.3 times faster at an input length I never send — Faster on the Benchmark, Slower Where It Mattered.

The calendar gets a vote

The second reason is less technical and mattered more on the day.

Earlier that same day an experiment of mine — described elsewhere, and entirely my own fault — had put the whole box into a kernel panic. Not a slow request, not a failed job: the machine stopped, and everything it serves stopped with it. Hours later I had a proven-correct change in my hand and an urge to deploy it before dinner.

Proven correct in an afternoon is not the same as proven safe on a machine that has already fallen over once that day. So it went behind a flag, switched off, pending an explicit decision made when I am not still cleaning up from the last one.

For a business: you have a change budget per day, and it is smaller than your appetite. After an outage, the right number of unrelated production changes is usually zero — not because the next change is bad, but because you have already spent the attention that would catch it going wrong. The failure that day, and the layered limits it produced, are in Can You Promise It Never Happens Again?

"Faster" and "identical" are two different products

The second switch is smaller with a sharper edge. It collapses roughly twenty-five tiny computation steps into one for every word the model produces. It works: decode went from 28.9 to 30.1 tokens per second at short context, 25.1 to 26.1 at medium, 23.5 to 24.3 at long. Call it 3.4 to 4.2 percent.

But the output stopped being bit-for-bit identical to the old behaviour. After about forty generated words, the two versions drifted apart. It was not a bug — combining the steps changes the order in which fractional numbers get added together, and that is enough to round differently. Both answers are equally valid. They are just not the same answer.

I left it off. Three or four percent is not worth surrendering the ability to say that today's system produces exactly what last month's produced.

The lesson for businesses: reproducibility is a feature with a price, and only you know whether you need it. If you ever have to re-run a case to defend a decision — a credit call, a claim, a triage outcome — an output that is "equally valid but different" is a very expensive four percent. If nobody will ever re-run anything, take the speed.

A switch left off is a decision, not unfinished work

The trap now is to leave two half-finished things lying around and forget which is which. So each gets a written condition that would flip it.

The first turns on when a workload arrives that is genuinely dominated by cold long-context work, in a scheduled window on a machine that has been quiet for a week. The second does not turn on at 3.4 percent — only if I ever run something where exact reproducibility stops being worth anything, and I do not have one.

The lesson: "we built it and it works but we haven't shipped it" is only a failure if nobody wrote down what shipping is waiting for. Record the trigger, and the shelved work becomes an option you can exercise later instead of a rumour someone re-argues every quarter. Deploying the moment a test goes green, without the trigger, is how a box ends up running nothing at all — Shipped Is Not Running.

The bottom line

Working and shipping are different milestones, and the gap between them is where judgment lives. A verified win still has to clear three questions: does it apply to our real traffic, is this a safe day to touch production, and does it change anything we promised would not change. I answered those honestly for two proven speedups and switched both off. Nothing was wasted — I now hold two options with known prices, which is more than I had that morning.

For the engineers

The first change is sparse attention that was implemented as masked dense. The model's attention is designed to attend to a small selected slice of history, so cost should stop scaling with context — but the implementation still computed the full history and then filtered it, discarding the saving the design exists to provide. Gathering only the selected blocks and computing those directly gives the same result: max absolute difference 0.0000 at the unit level, and identical timings below the 24,000-token threshold where the sparse path is not taken at all. Cold 100,917-token prefill: 648.5 s at 155.6 tokens per second, versus 313.5 s at 321.9. Per chunk it scales with context — 1.34 times at 32k, 2.44 at 65k, 3.31 at 98k, where 23.4 s per chunk becomes 7.1 s. Memory flat, no swap, no compressor activity.

The second is kernel fusion in the decode path. That step was roughly twenty-five separate small operations per token, each paying about 25 microseconds of fixed launch overhead — far more than the actual data movement justified. Fused: 0.675 ms versus 0.768 ms per step, 3.4 to 4.2 percent on end-to-end decode. It is not bit-exact; floating-point reduction order changes and sampled output diverges after about forty tokens. Both live behind flags, both default off. The first is a deployment decision waiting on a maintenance window; the second is a determinism trade I declined.

If your team has optimisations that were benchmarked green and shipped the same afternoon — or the opposite problem, proven improvements nobody dares switch on because the criteria were never written down — that is a consulting day spent on judgment rather than code, for your business in Dubai, the UAE, or anywhere your AI decisions get made. I will go through what you have measured, separate "this works" from "this helps us", and leave you with explicit switch-on conditions for the ones worth keeping. Book a consulting day or send me an inquiry and I'll help you write the switch-on conditions.