Two Hundred Seconds, One Shuffled List

For weeks, the first question I asked my own AI assistant each morning took just over three minutes to come back. The second question took seconds. So did the third, and every one after it, until I closed the window and opened a new session — at which point the three minutes came back.

I had accepted this. It felt like the cost of running a large model in my own office instead of renting one: things warm up, you wait, you get on with your day.

It was not the cost of anything. Measured properly, the first message took 198 to 200 seconds, every single time, and roughly 99.94% of that work was work the machine had already done and could have skipped. The reason it did not skip it was a list — the same list, with the same items — that arrived in a different order on every session.

The model was never the slow part

Here is the shape of the problem, without the machinery.

Before my actual question reaches the model, the assistant sends a large block of standing context: instructions, the definitions of the tools it can use, a menu of the capabilities available to it. In my setup that block ran to about 26,000 tokens — call it a briefing document that gets read out in full before anyone is allowed to ask a question.

The model has to process that briefing before it can answer. That is what the 200 seconds were. My actual question, once I finally got to ask it, was about 16 tokens. Sixteen. The other 26,000 were preamble, re-read from scratch, every morning, for content that had not changed since the day before.

Every serious inference stack has a fix for this: it keeps the processed briefing and reuses it. Mine had that fix. It was switched on, correctly configured, and doing nothing.

A cache cannot tell that two identical lists are identical

The reuse mechanism matches on an exact prefix. It compares what you sent this time against what you sent last time, byte for byte, from the beginning, and reuses everything up to the first difference. Everything after that first difference has to be redone.

That design is not a flaw. It is the only cheap way to do it — the comparison has to be faster than the work it saves, so it cannot be clever about meaning.

Somewhere near the top of my briefing block sat a list of available capabilities. Same items, same wording, same length, every session. Emitted in a random order every session. To a human reading a diff, those two lists are the same list. To a byte-exact prefix match, the first swapped line is where the two documents stop being the same document — and everything downstream of it, the entire remaining twenty-something thousand tokens, is unrecognisable.

One unstable field, early in the payload, silently defeated more than 99% of a cache that was otherwise working perfectly.

The failures that never raise an error are the ones that bill you

Nothing was broken. That is the part worth taking away.

No error was logged. No warning appeared. No health check went red. The assistant worked, the model answered, the answers were good. The only symptom was a number — 200 seconds — that lived in the gap between "slow" and "broken", which is exactly the gap where problems go to live for months.

For me the penalty was time. On a metered cloud API, the identical defect arrives as money: providers discount reused context heavily, so a payload that misses the cache on every request is billed at full rate on every request, and the line on the invoice looks like ordinary growth. If you run anything at volume against a hosted model, cache hit rate is a financial metric, not an engineering one. Ask for it by name. Most teams cannot produce it, which is itself the answer.

Fix it in the layer you control

The obvious repair was to change the tool that was emitting the shuffled list. I did not do that.

That tool is not mine. It updates, frequently, and a private fork of someone else's fast-moving software is a maintenance bill that arrives forever. Instead I put the fix in the proxy layer that already sits between the assistant and the model — the piece I own. It sorts that list into a stable order before the request goes anywhere near the model. The assistant stays stock. The model sees the same bytes every time.

The general form of that decision matters more than this instance: when a vendor's output is almost right for you, normalise it in a layer you control rather than forking the vendor. Adapters are cheap to maintain and easy to delete. Forks are neither.

Verify with the counter, not with the stopwatch

The fix looked like it failed at first. The next session still took the full 200 seconds — because the cache now had to store one canonical, sorted copy before it had anything to reuse.

The session after that reprocessed 1,437 tokens out of roughly 26,000 — 95% reuse. The one after that reprocessed zero. Steady state settled at about 16 new tokens per fresh session — my question, and nothing else — and the first message dropped from 198–200 seconds to 10–15 seconds of wall-clock time.

I only knew any of this because I was reading the reuse counter rather than watching a clock. A stopwatch would have told me the fix did not work, and I would have reverted it. If you are paying for a cache, instrument the thing the cache reports about itself.

The bottom line

The cache was fine. The configuration was fine. The model was fine. The system was being fed one field that changed for no reason, and that was enough to throw away almost all of the benefit, quietly, with no error to investigate. A cache built on exact matching is only as stable as the least deterministic thing you put into it — and the cost of that instability shows up as latency you tolerate or a bill you assume is normal.

For the engineers

Automatic prefix caching in a local inference server keys on byte-identical prefixes of the tokenised prompt. There is no fuzzy match and no per-field awareness: the server walks the incoming token sequence against cached entries and reuses up to the first divergence. A ~26,419-token system block contained a capability list the harness serialised in non-deterministic order per process start. One transposed line shifts every subsequent token position, so the longest common prefix collapsed to whatever preceded that list, and the server re-prefilled essentially the whole thing — 198–200 s at the prefill throughput available on that hardware.

The fix is a pre-call hook in the proxy that normalises volatile fragments before the request is forwarded: sort the list deterministically, then hand it on. Two details cost me time. First, sort on the entire line, not on a prefix key — several entries shared a namespace prefix and collapsed into the same key, giving a stable-looking sort that still permuted. Second, the hook is loaded from the proxy's own configuration directory, not from the interpreter path, and edits do not take effect until the proxy process is restarted, so a correct fix can read as a failed one.

Measured after priming: ~16 tokens prefilled per fresh session out of ~26,000, 99.94% reuse, first-message latency 10–15 s. The intermediate run showed 1,437 tokens reprocessed — 95% — before settling at zero.

The other half of this lesson — that a cache living only in memory is a loan, not an asset — is in The Cache That Vanished Every Restart, where I made this same cache survive restarts. And on the money side, The $370 AI Bill: A Cost Audit Post-Mortem is what this defect looks like when it arrives as an invoice instead of a delay.

If your AI features are slower or more expensive than the vendor's published numbers suggest and nobody on the team can tell you why, that is usually not a model problem — it is a payload problem, and it is invisible from the dashboard. That is the shape of a useful consulting day: I'll instrument what you are actually sending on every request, find the fields that change when they shouldn't, and leave you with the hit-rate number your invoice depends on. Book a consulting day or send me an inquiry first if you'd rather talk before booking.