Five Seconds of Dead Air

You type a short question into a model running on a machine three metres away, press enter, and watch a cursor blink at you. Nothing happens. Then, after an uncomfortable pause, words start arriving fast and the answer is correct — the way it has always been correct.

I had no idea why it took five seconds before it started thinking, even when the conversation was already cached. A cache exists so that repeated context costs almost nothing. Short question, warm cache, instant answer: that was the whole point of the thing.

So I measured instead of guessing. On those cached requests, the model's own log said it had spent 2.5 seconds doing the computation. The wall clock said 8.3. Nearly six seconds, on every single request, doing something nobody had asked for and nothing was reporting.

Nothing was broken, and that is exactly why it survived

There was no error. No failed request, no alert, no retry, no line in a log marked with a warning. Every answer that came out was the right answer. If you had audited that system on correctness alone — which is what almost every test suite I meet actually audits — it would have passed cleanly.

Correctness and speed are separate properties of a system and they degrade independently. Normal use tests one of them constantly and the other one never. A wrong answer generates a complaint on the day it appears. A slow answer generates a shrug, then a habit, then a belief: that's just how it is with this thing.

The lesson for businesses: your automated checks almost certainly confirm that the output is right. Very few of them assert that it is still as fast as it was in month one. Nothing in your stack will ever page you because something got slower.

The system's own stopwatch was measuring the wrong thing

The reason this hid so well is that the instrumentation was honest and useless at the same time. The model reported how long it spent reading the prompt — 2.5 seconds — and that number was accurate. It simply had nothing to say about the six seconds spent before that clock started.

Every dashboard measures the work somebody expected to be expensive. The overhead nobody anticipated sits outside the instrumented span by definition, which is precisely why it can grow without limit. My timings were not lying. They were answering a narrower question than the one I thought I had asked.

When someone shows you a performance graph, the question worth asking is not "what does this number say" but "what happens outside this measurement" — because the gap between the wall clock and the reported figure is where the problem lives.

The requests that should have been free paid the most

The cost was a fixed toll, and fixed tolls fall hardest on small jobs.

For prompts under 15,000 tokens the model computed its part in 0.33 seconds. Time to the first word on screen was 4.5 seconds. More than thirteen times as long waiting as working, on exactly the quick, cheap interactions you most want to feel instant.

This is worth translating, because it is the same shape as a business process. A fixed approval step that adds two days is a rounding error on a six-month project and a catastrophe on a one-hour task. If you have automation where small requests feel disproportionately sluggish compared with big ones, you are almost certainly looking at a fixed overhead, not a slow engine — and fixed overheads are usually the cheapest thing in the entire system to remove.

The thing that slowed it down was the thing that was working

Here is the part I find genuinely instructive. The cache did its job well, so it filled up. Two days of long working sessions grew the store on disk to 1,005 saved entries, roughly 960 GB of them.

And on every request — every one, including a two-line question — the lookup opened all 1,005 of those entries and re-read their contents from disk to decide which one to reuse. Around 31.6 million numbers parsed, from scratch, before a single word of the answer was produced.

That code was not wrong. It was correct at the size it was built and tested against, which was a few dozen entries. Nobody wrote a bug. Success moved the system into a size range its design had never been asked about, and the design answered the same way it always had, only slower each day.

The lesson for businesses: the component most likely to be quietly failing is the one that has been most heavily used. Growth is the load test you never scheduled, and it arrives gradually enough that nobody notices the day it crossed a line.

Some speedups buy time by giving something up. This one did not.

My own next question, before shipping anything, was whether the fix could lose anything or whether it was a genuine optimisation. That distinction matters more than the speed number, and I would push any vendor or engineer hard on it.

Some speedups buy time by giving something up — precision, completeness, a check that used to run. Others simply stop doing work that was never needed. This one was the second kind: the answer came from data already sitting in memory instead of being rebuilt from disk on every call. Same decision, same selection, same result.

I proved that rather than asserted it. Before it went anywhere near production, the new path was tested against the real 960 GB store on eight cases and had to pick the byte-for-byte identical entry every time. Then it shipped the same day. That "proved and shipped same day" instinct has a limit, which I explore in The Speedup I Built and Refused to Ship.

The lookup that had been costing seconds now takes between one and three milliseconds. A real call from a client's triage system, measured end to end over the network, went from about 4.5 seconds to 0.66.

The bottom line

A system that gives correct answers can be getting steadily worse at giving them, and nothing about correctness will tell you. Slowness does not raise alarms; it lowers expectations. The five seconds had been there long enough that I had almost stopped seeing them, and the fix — once I actually looked — took one day and cost nothing. The cache in this story had also outgrown a count-based limit in an earlier life; the arithmetic of a store that fills past its design is in Two Parts That Work, Breaking Each Other.

For the engineers

The lookup was longest-matching-prefix over a disk-backed prompt-cache store: given an incoming prompt, find the saved KV-cache entry whose token sequence is the longest prefix of it. The implementation re-opened every entry's header on disk and re-parsed its full token-id list per request — about 1,005 entries and 31.6 million integers, in interpreted Python, with a four-entry cache in front of the file headers, so in practice nearly every read missed.

The symptom: median wall-clock time inside the prefill phase 8.3 seconds against 2.5 seconds of logged compute. Under 15,000 tokens, 0.33 seconds of computed prefill against 4.5 seconds to first token. The scan was O(store size) and independent of prompt size, so it dominated every small request on the box.

The fix is unglamorous: parse the index once at startup, hold it in memory, keep it current on write, and answer the same query from the in-memory structure. No change to the selection rule, no change to eviction, no approximation. Validated against the live 960 GB store on eight cases for identical entry selection before deployment. Lookup now 1–3 milliseconds; end-to-end client request 4.5 seconds to 0.66.

If nobody owns the latency number, this class of bug is permanent. It never errors.

If something in your business used to feel quick and now feels sluggish — and everyone has quietly accepted it, because the output is still correct and nothing is technically broken — that is a very good use of a consulting day, whether you're in Dubai, across the UAE, or anywhere in between. I will measure where the time actually goes, separate the overhead you can delete from the work you cannot, and tell you plainly which is which. Book a consulting day or send me an inquiry and name the thing that used to feel quick.