Can You Promise It Never Happens Again?

The benchmark was, in my own description, harmless: single layers, minimal impact, a few minutes on a machine with room to spare. Ninety-four seconds later the kernel's own watchdog had stopped receiving check-ins, and the machine in my office was a brick — deaf to the keyboard, recoverable only by cutting its power.

Three production model servers went down with it.

The crash log had one line worth framing: the memory compressor at 100% of its segments limit, across six swap files. The machine had not run out of memory the way people picture it. It had run out of the ability to shuffle memory around — worse, because at that point even the watchdog whose job is to notice trouble cannot get a turn on the processor.

Then comes the question every client asks after an outage. Can you make sure this never happens again?

"Never again" is a promise nobody can keep

No. I cannot promise that, and neither can anyone selling it to you.

What I can promise is more useful: it fails as one process dying instead of the whole system going down. That commitment is testable, and it changes what the bad day costs you.

The goal moves from prevention to blast radius: stop building the one perfect check, build independent limits that each catch what the one above missed, then write down what none of them cover. A fix delivered with an honest list of its gaps is the stronger fix.

Why that machine was a bomb waiting for a timer

I want to be precise about what "room to spare" meant, because the benchmark only lit a fuse that had been sitting there for weeks.

The machine is an M3 Ultra Mac Studio with 512 GB of unified memory, and the number on the box is not the number you get. Model weights are pinned and never move; on this box, the memory actually available for live work was roughly 50 GB after the fixed costs. Sitting on top of that remainder was a cache budget of 60 GiB — larger than the entire headroom it was meant to live in — and a concurrency limit of 25 requests, each needing its own slice of the same space.

Every setting was defensible alone; none had been checked against the others. A budget larger than headroom is not a limit — it is a timed bomb, and the benchmark was the timer. The full arithmetic of that morning is in One Workload Too Many.

The cache taught me the same lesson from a second door: it was first capped by item count — 24 saved snapshots — and came back anyway, because 24 near-duplicates of a single very long conversation filled about 73 GB. A cache bounded by item count is unbounded in bytes; the cap that held was 8 entries against a hard 48 GB byte budget. "We limit it to 24" is not a limit when the cost per item varies by more than a factor of ten. Bound things in the unit that actually runs out.

The alarm everyone trusted had never once fired

Before rebuilding anything, I replayed the existing alarms against history. The long-standing rule — alert when free memory drops below 5% — had fired exactly zero times across the entire bad stretch. Zero. A newer rule watching how full the memory compressor was accounted for 102 legitimate kills over the same period. Nobody had noticed, because a silent alarm and a healthy system look identical from outside.

An alarm that has never fired is not proof that nothing went wrong. Ask for the count: a monitor that has triggered zero times across a period you know contained incidents is not protecting you, it is decorating your dashboard.

Current slack proves a control is working, not that it's unnecessary

The mirror image of the dead alarm is the limit that looks pointless, and I nearly relaxed one for exactly the wrong reason.

Days earlier I had been shown a very good argument for loosening the entry cap on that same cache: in a single day it had hit its ceiling and thrown work away 331 times, while the byte budget attached to it never rose above a quarter full. Slack everywhere — surely the count cap was redundant.

What saved me was opening the incident history instead of re-arguing the current numbers. Under the looser configuration — 24 entries against a 96 GB budget — the cache had stayed comfortably inside its byte ceiling on the day of an earlier lockup, and the model was still watchdog-killed 61 times in a single day, with 104 kills across the bad stretch as a whole. Since the tighter 8-entry cap went in: zero kills. The two limits were catching different failure modes, and only the incident record showed it. When a control looks redundant, the honest question is not "what does today's slack say?" but "which failure was this added for?" — and that answer is in the logs, not on the dashboard.

A guard that kills the wrong process is worse than no guard

An earlier run of the same benchmark had not taken the machine down — but it triggered two automated kills of an unrelated production service, and I read those kills as evidence that the service itself was failing. I spent time on the wrong suspect.

The guard chose its victim by "whichever heavy process started most recently," not by "whichever process is actually eating the memory." It shot a bystander twice while the real offender grew.

That is what makes a bad guard dangerous rather than merely useless: it manufactures a false incident, points the investigation at an innocent component, and buys the real problem time.

Four layers, and an honest list of what they still miss

The rebuild was four independent fixes.

• A rule about me: no memory-heavy experiments on the production machine outside a maintenance window, and never launched by a script that thinks the machine looks idle. I have broken my own safety rules before under mild pressure — I Broke My Own Safety Rule — so this one is written down, not remembered.
• A wrapper that gives any experiment a hard memory ceiling and kills it the instant it crosses. Verified, not assumed: it killed a process at 2.8 GB against a 2 GB budget.
• The rebuilt guard — a second version, the first having been written after an earlier lockup — which trips on any swap usage above 256 MB instead of waiting for compressor occupancy above 96 GB, and kills the largest consumer rather than the newest arrival.
• The operating system's own per-process limit, which I keep and do not rely on — by the time it fires, the machine is already thrashing.

What those four still do not cover: a fault below my level, in the operating system itself, and a spike fast enough to outrun the guard between checks. I say that out loud when I hand the work over. A client who knows the gaps can price them; a client told "it's handled" finds out the hard way.

You do not need the root cause to shrink the blast radius

Later the same day I caused a second crash, unrelated to the first. Testing an optimisation, I restarted the same large model process three times in under two minutes — kill, wait, force it, respawn — each cycle re-mapping over a hundred gigabytes of files. A kernel counter tracking those mappings overflowed, inside a process idle for the previous two minutes. I still cannot establish that root cause from the logs available.

But the crash was not the expensive part. What the unclean reboot left behind was.

A stale lock file. The database underneath my API layer records its own process number on disk; that file survived the crash, and on the next boot the operating system had given the same number to an unrelated service. So the database read the lock, decided a copy of itself was already alive, and refused to start. I had watched this exact mechanism take down a service once before and had not generalised the fix. This time the proxy crash-looped through 33 failed startup attempts, and every client was offline for 20 minutes until I deleted the file by hand.

None of that chain required understanding the kernel bug. The fix was a few lines at startup: before trusting the lock file, check that the process it names is actually a database; if not, delete it and carry on. That outage now heals itself at boot.

Waiting for a complete explanation before hardening anything leaves the cheap fixes on the table — and the cheap fixes are usually where the downtime lives. The inverse mistake — shipping the fix before the machine has had time to prove itself again — is Shipped Is Not Running.

The bottom line

"Never again" is a sales answer. The real answer is a stack of independent limits, each catching what the one above it missed, plus a written list of what is still not covered — and limits set in the units that actually run out, checked against real headroom rather than the number on the box.

Ask whoever built your system which single failure takes the whole thing down, and how long it stays down before anyone notices. Twenty minutes of mine was a leftover file with an out-of-date number in it.

For the engineers

The inference framework's own memory limit only raises after RAM and swap are both exhausted — post-thrash, useless as prevention. The prior guard selected its victim by newest heavy daemon rather than resident size, killing a bystander twice while the offender grew. The redesign changes both: it fires at the earliest reliable signal — any swap usage above 256 MB, where it previously waited for compressor occupancy above 96 GB — and selects culprit-first by footprint. Historical replay: the 5%-free-memory alarm fired zero times across the bad stretch; the compressor-occupancy rule accounted for 102 legitimate kills. The budget wrapper was verified killing at 2.8 GB against a 2 GB ceiling. Panic signature: watchdog timeout, no check-ins for 94 seconds, compressor at 100% of segments limit across six swapfiles.

The second panic was a page-mapping reference-counter overflow after three forced kill-and-respawn cycles in two minutes on a process mapping over a hundred gigabytes, idle at the moment of the panic, not root-caused from userspace logs. Its downstream damage was understood: a stale process-ID lock file survived the unclean shutdown, its ID was reused by an unrelated service on the new boot, and the database refused to start — proxy crash-looping 33 times, every client offline for 20 minutes. The startup guard now validates that the process named in the lock file is a running instance of that database before honouring it, and unlinks it otherwise.

If your system has one machine, one process or one leftover file that takes everything down with it — or someone has told you an outage "cannot happen again" without showing you how — that is a consulting day I would take seriously, whether you operate in Dubai, across the UAE, or somewhere I never visit at all. I will trace what shares a machine with what, replay your alarms against your incident history to find the ones that have never fired, and leave you with limits that kill one process instead of the whole box, plus an honest list of what they still miss. Book a consulting day or send me an inquiry if somebody has already made you that promise and you want it examined.