The Cache That Vanished Every Restart

There is a kind of waiting I had stopped noticing. I restart the model server in my Dubai office — an update, a config change, a crash, a power blip — then send the first message of the day and go make coffee, because the machine is busy re-reading everything it already read yesterday.

The system was fast. It was fast because it kept a cache in memory of work it had already done. And every restart threw all of that away and paid the full bill again from zero.

When I finally measured both ends, the number was not subtle: the same request cost 24.1 seconds cold and 4.5 seconds warm. A 5x difference, handed back to the void every time a process died.

A cache that lives only in memory is a loan, not an asset

This is worth translating out of engineering, because it shows up in far more places than AI.

A cache is a store of finished work — you paid for something expensive once and kept the result so nobody pays again. It is one of the highest-return things in any system. But if that store lives only in the machine's memory, it is borrowed, not owned. It exists exactly as long as the process does. Restart, crash, update, and it is gone — not degraded, gone — and you are back to first-day performance with no warning to anyone.

What makes it dangerous is that it is invisible on a good day. The dashboards show a fast system. Nobody measures the first minute after a restart, because restarts are rare and nobody is watching at 3am. The cliff is real, it just isn't on the report.

Measure the cliff before you pay to remove it

I did not start by building. I started by working out what the durable version would cost to run, because "add persistence" is the kind of decision people approve on instinct and regret on the invoice.

The resource was disk. I set aside up to 4TB for the feature, from a much larger free pool, then modeled the realistic sustained usage for my actual pattern of work. The answer came back at roughly 500GB to 1.5TB.

That gap is the whole lesson. The ceiling I was willing to fund was several times the volume I would really use — which turned a leap of faith into an easy yes, and set the cap high enough that the feature would never quietly strangle itself in six months.

The lesson for businesses: before you buy durability, size it twice — once for what you would tolerate at the worst, once for what you will realistically consume. If you only ever produce one of those numbers, you are either overpaying or about to be surprised.

The rule that mattered most was not about speed

The obvious design goal was "make the fast thing survive a restart." The goal I spent the most care on was making it impossible for the cache to be wrong.

A cache that occasionally serves the wrong answer is far worse than a slow system, because the wrongness is silent and arrives with full confidence. My specific hazard was models: I run more than one, I switch between them, and cached work from one model is meaningless — and misleading — to another.

So every stored entry is keyed to a fingerprint of the exact model weights that produced it. Different model, different key, no possible collision. And I shipped the whole feature off by default, so nobody inherits new failure modes they did not ask for.

The instinct worth copying: when you make something faster, spend your paranoia budget on the ways it could become quietly incorrect, not the ways it could become slow. Slow is visible. Wrong is not.

Set the overhead budget before you write the code

Persistence is never free — you are adding work to the fast path to protect a slow one. So I decided in advance what I was willing to pay: no more than 10% overhead on normal requests. Measured, the finished version costs about 5–7%.

That number existed before the code did, which is why it was a constraint rather than a rationalization after the fact.

Two designs got rejected on the way. The first only survived planned reboots — I threw it out because the failures I care about are crashes, and a safety net that only works when you fall on purpose isn't one. The second was a more elegant scheme where entries shared structure to save space; too complicated to trust in a first version, so each entry is now self-contained and boring. Boring survives.

Give the fix back to the tool instead of hoarding it

The whole thing was built overnight. I wrote a specification and a task-by-task plan, then ran the implementation autonomously while I slept — each task isolated, and every task required to write a failing test before writing the fix.

The morning produced 30 commits across 13 files — 3,314 lines added, 12 removed — with 64 unit tests and 3 full end-to-end tests. The unit suite runs in about 7 seconds; the end-to-end suite takes about 167 seconds because it genuinely starts servers and restarts them.

Then I sent it upstream to the maintainers of the open-source inference server we run — shutdown bug and all, written into the covering note — instead of keeping it as a private patch on my machine.

That was a business decision, not a charitable one. A private patch is a liability you re-apply and re-test after every upgrade, forever, owned by exactly one person. Upstreamed, it gets reviewed and carried forward by people who are not me. The cost was writing it to a standard someone else would accept. The return is never maintaining it alone.

The bottom line

If your system is fast because something is held in memory, you do not have a fast system — you have a fast system between restarts. Making it durable is real work: a design, a budget, tests, honesty about the edges. But when the measured payoff is a 5x latency drop that survives a crash, it is worth doing properly and worth giving back to whatever tool you built it on. The speed was never the hard part. Keeping it was.

For the engineers

The design is a write-through, off-by-default disk tier sitting behind the existing in-memory cache. Three properties do the actual work.

First, entries are written asynchronously immediately after they are produced, so the request path never blocks on disk I/O — that is where the 5–7% overhead lives, comfortably under the 10% budget.

Second, every write is atomic: write to a temporary file, then rename. A crash mid-write can never leave a torn or half-valid entry behind. Worst case, one in-flight write is dropped and the previously renamed file remains correct.

Third — the one that is easy to miss — an entry's on-disk last-used timestamp is refreshed even when the hit was served entirely from RAM. Without that, your hottest entries look untouched to the disk-eviction policy and get deleted precisely because they were too popular to reach disk.

On startup, the process rebuilds its full index from what is on disk before accepting traffic, so a freshly launched server immediately reuses work from a process that died hours or days ago. Keys are per-model, derived from a hash of the weight manifest, which makes cross-model contamination structurally impossible rather than a matter of discipline. Verified in production: 24.1s cold, 4.5s warm, with 15 of 16 prompt tokens served from disk rather than recomputed.

I shipped that disk tier with a known shutdown bug, on purpose — Shipping With a Bug You Can Name explains the reasoning. And the quieter half of the caching story, where one unstable field defeats the whole cache without restarting anything, is Two Hundred Seconds, One Shuffled List.

If your product is quick because of something warm in memory — a cache, an index, a loaded model, a session store — and nobody has ever measured what the first sixty seconds after a restart costs your customers, that is one consulting day and one clear question: I'll measure the cliff on your system, tell you whether closing it is worth the money, and if it is, leave you with a durable tier that survives crashes instead of only surviving polite shutdowns. Book a consulting day or send me an inquiry first if you'd rather size the problem before booking.