The Status Line That Cost Me the Cache

I made a commit. Then I opened a fresh session with the coding assistant I use every day, in the same project folder, seconds later, and asked a question. It took about twenty seconds before the first word came back.

Nothing had changed that should have mattered. Same machine, same folder, same model loaded and warm. The only event between the fast session and the slow one was a commit — a few files, a one-line message, work I had already finished.

That commit was enough to fork a cached copy of a roughly 40,000-token block of standing context, 86% of which was a list of tool definitions that had not changed in weeks — 155,062 characters describing 69 tools, re-filed as new material because five lines of commit subjects further down the page were different.

Somebody decided freshness was worth more than the cache, and it wasn't me

Before my question reaches the model, the assistant sends a large standing block: instructions, definitions of every tool it can use, some project context. Systems keep a processed copy and reuse it rather than pay twice — and this was the second time something unstable had been buried in mine.

Inside that block, for any folder that is a git repository, the assistant helpfully includes a live status: which files are currently modified, and the last five commit messages. Useful. It also changes every time I save a file and every time I commit.

A cache that matches on exact content cannot tell the difference between "five lines moved" and "an entirely different document". Each commit produced a new variant of the standing block, and each variant became its own stored copy. In one active project repository, four commits landed in about two hours — four copies of the same 40,000-token preamble, differing by a handful of lines, competing for the same finite space.

The lesson for businesses: every piece of live status you inject into a cached context is a purchase. You are buying freshness with cache efficiency, and the exchange rate is terrible — a few lines of recency for tens of thousands of tokens of re-processing. That is a defensible trade if you make it deliberately. Almost nobody does. This is the same instinct that runs up the bill in The $370 AI Bill: A Cost Audit Post-Mortem — the charge arrives looking like ordinary growth.

The expensive part looked like the stable part

What makes this hard to catch is that the block looks stable. Describe it in a meeting — "it's the tool definitions and some project context, it never changes" — and everyone nods, and everyone is 86% right.

The other 14% is where the money goes.

At volume against a hosted model this arrives as an invoice rather than a delay. Providers discount context they have already processed. A payload that fragments into a new variant every few minutes is billed at full rate every few minutes, and on the bill it looks like ordinary growth.

For a business: audit the contents of the thing you labelled "static", field by field. The parts of a request nobody thinks about are the parts nobody has checked.

Ask what the freshness is for before you pay for it

The question I asked before touching anything was not "how do I make this faster". It was: if I remove the live git status, do the model's answers get worse?

They do not. Both of the fields I stripped — the list of recent commits and the modified-file status — are one cheap tool call away from being re-derived the moment the assistant actually needs them. Nothing was lost; it moved from "pushed into every request forever" to "fetched on demand, when it matters".

So I stripped exactly those two, and kept the fields that are genuinely stable: the branch name, the default branch, the author. Branch names change a few times a week. Commit lists change a few times an hour.

The rule to take away: separate what is genuinely volatile from what merely looks dynamic, and keep the first out of the cached part. Anything the system can fetch when it needs it does not belong in the preamble of every request.

Your context is less deterministic than your diagram claims

While measuring this I found a second cause underneath the first, quieter and worse.

The number of tools in that block varied between sessions on the same machine, in the same folder, minutes apart: 34, then 36, then 38, then 65, then 69. Nothing in my setup had changed. Background integrations connect asynchronously after the assistant starts, so whichever ones had finished connecting before I typed determined the contents of the block — including an undocumented intermediate state that appears mid-connection.

The largest, most expensive, most stable-looking section of every request was being assembled by a race condition. Send your first message a second earlier and you get a different document.

The lesson for businesses: if the shape of your request depends on start-up timing, you do not have a stable prefix, you have a lottery. Anything that assembles context from several services at start-up has the same exposure.

Hash what you actually sent, not what you meant to send

I did not deploy the strip rule and hope. I ran it against real captured requests first, and the test was blunt: two standing blocks from two sessions that should have been identical, and were not, collapsed into one identical block after stripping.

Combined with the other fixes I made that day, token-weighted cache reuse landed at 79.0%, up from close to nothing on the pattern I had been hitting. I cannot attribute all of that to this one change — several things moved at once — and I am not going to pretend otherwise. What I can say is that two prompts that differed before were identical after, which is the only thing a cache of this kind cares about.

The bottom line

Nothing here was broken. The assistant worked, the answers were good, the cache was correctly configured and doing its job on the material it was given. It was handed a fresh document every few minutes and treated it as one, because that is what it was. Live status in a cached context is a trade — recency for money — and it is common because nobody remembers making it.

The sequel to this story came days later, when the cache itself turned out to be only half the contract: Two Parts That Work, Breaking Each Other covers what happened when a new model's output shape didn't fit the caching code around it.

For the engineers

The assistant renders its full tool-definition list first: 155,062 characters, 69 definitions, about 86% of a ~40,000-token prompt. Immediately after it comes a system block that, for any git working directory, embeds live git status output and the last five commit subjects.

Prefix caching matches byte-exact. The tool block preceding the git block is technically still reusable, but every commit and every dirty-file change mints a new variant of the full standing block, each landing as its own cache entry. Four commits in roughly two hours produced four distinct lineages of the same preamble, all evicting each other from a bounded store.

Compounding it: the tool list itself is nondeterministic at session start. Background tool integrations connect asynchronously, and the count observed across sessions was 34 / 36 / 38 / 65 / 69, plus a third, undocumented placeholder state that appears mid-connection. Same host, same directory, different document.

The fix was a fifth text-stripping rule in the request-processing proxy — the layer I own, so the assistant stays stock — removing only the recent-commits list and the dirty-file status, while preserving branch name, default-branch name and author. Validated against captured production prompts: two previously distinct system prompts hashed identical afterwards. Token-weighted reuse for the day settled at 79.0%. No functional loss: both stripped fields are one cheap tool call from being re-derived on demand.

If your AI product injects live context into every request — a status line, a recent-activity feed, the user's last five actions, a timestamp — you are probably paying full price for context you were promised a discount on, and nothing in your dashboard will say so. I ran that audit on my own stack and turned a near-zero cache hit rate into a measurable one. If you're running AI features that cost more than they should — in Dubai, across the UAE, or anywhere your invoice arrives — a consulting day is exactly this: I'll capture what you are actually sending, separate the fields that must be fresh from the ones that only look dynamic, and hand you back a cache hit rate you can put next to the invoice. Book a consulting day or send me an inquiry first if you'd rather talk before booking.