One Workload Too Many

The machine that runs my AI work is configured never to sleep. That is deliberate — things depend on it answering at any hour, and a server that naps is a server that drops connections. One afternoon it went dark anyway.

Not crashed in the way software crashes. No error, no restart, no line in a log. It simply stopped answering: no remote session, no way in from another machine, nothing left to ask politely. I reached behind the desk and held the power button, which is the least sophisticated tool in the building and the only one that worked.

The cause was arithmetic. At the moment it locked up, I was asking that machine for roughly 600 GB of model memory. It had roughly 550 GB to give. Nothing in the chain warned me, nothing refused the request, and nothing stepped down politely. It starved, and then it went quiet.

The workload that breaks you is never the one you are adding

I was midway through deploying a new model layout, which meant four things were resident at once. A new model, fully loaded, at about 180 GB. A second model, still loading, at about 150 GB. A third model I had finished with, holding about 100 GB. And a supporting service holding about 64 GB. Free memory bottomed out somewhere between 0.1 and 0.3 GB. Call it nothing.

I knew about the first two. I had budgeted for them and the sum worked. What broke the budget was the pair I had already written off in my head.

The lesson for businesses: capacity accidents are almost never caused by the thing you are adding. They are caused by what you believe you have already released.

"Idle" is a state of mind, not a measurement

I had stopped sending work to that third model about fifteen minutes earlier. In my mental model it was parked and costing nothing. On the machine it was holding about 100 GB, against roughly 37 GB it actually needs to exist. The supporting service was holding about 64 GB against roughly 15 GB it needs.

So about 110 GB — a fifth of the machine — was software sitting on memory it had finished using, because nothing had ever told it to hand any back. It had been busier earlier in the day, it had grabbed what it needed then, and it never let go.

"Not currently in use" and "not currently costing you" are two different statements, and most software quietly assumes you meant the first. You have seen the same shape on a cloud bill: the environment nobody has opened in three weeks, still provisioned, still charging. Nothing is wrong. Nothing is being used, either. That is the audit in The $370 AI Bill: A Cost Audit Post-Mortem, one layer deeper.

Your platform's safety net may not reach the biggest thing on it

When a computer runs short of memory it has two defences. First it compresses what it can, to buy room. Then it starts shutting down the least important programs until it recovers. Both defences worked exactly as designed here, and both were useless.

Model memory is held for the graphics processor, and that kind of memory cannot be compressed or set aside on disk the way ordinary memory can. So the system squeezed everything else until there was nothing left to squeeze, then went looking for programs to shut down — and the only ones it was allowed to touch were the small background services. The four large processes actually consuming the machine were never candidates. It removed the office plants while the building filled with water, and then it drowned.

For a business, the point is not the mechanism. It is that every platform's promise of graceful degradation has a boundary, and the workloads that matter most are often the ones sitting outside it. Ask your team which of your systems the automatic protections cannot touch. That is the list where a small overcommitment becomes a total outage instead of a slow afternoon.

Check what is resident, not what you remember

The rule now is dull and absolute: nothing large starts until I have looked at live free memory and compared it against a written budget for the thing I am about to start. It takes seconds. I had that habit for disk space and never built it for memory, because memory felt like something I was already tracking. I was tracking a story about memory.

The business version is the shared budget, and it fails the same way. Before another AI workload goes onto a shared machine, cluster or line item, ask for the current resident figure — what is consumed this week — not what somebody freed up on a whiteboard in March. A number given from memory is always the number from the day someone last cared to look.

Something has to be allowed to say no

Two changes came out of this. A small watchdog now samples free memory and, if it drops under a threshold, shuts down the most recently started model server. Newest first, because the newest arrival is the change that has not yet proven itself, and killing it returns the machine to a state that worked an hour ago. And each server is now capped on how much memory it may hold on to after it is finished with it.

Neither is a cure, and I want to be honest about that. The watchdog is a controlled amputation — it protects the machine by killing something I asked for. The cap costs a little speed, because memory a program cannot keep, it has to ask for again. I took both trades, because a slightly slower machine is a working machine and an unresponsive one is a drive across town to press a button.

The guard itself needed hardening later, when it turned out a guard that kills the wrong process can manufacture a false incident — that story, and the honest limits of "never again", are in Can You Promise It Never Happens Again?

The bottom line

The machine did not fail because one model was too big. It failed because four reasonable things were resident at the same time and two of them were things I was certain had gone. Software will let usage creep right up to the edge and then break catastrophically rather than gracefully — it has no incentive to warn you, and nobody wrote the part that says no. Measure what is actually resident before you add one more thing. The check costs seconds. The alternative costs a power button.

For the engineers

Two failures stacked. First, unified-memory model weights and KV cache are GPU-resident and effectively unpageable, so when demand exceeded physical memory the usual defences could not help: the compressor ran until it was exhausted, free memory sat at 0.1–0.3 GB, and the low-memory killer could only reap small-footprint background processes. The multi-hundred-gigabyte inference servers causing the shortage were never viable candidates. The system did not panic and it did not shed the right load — it starved into total unresponsiveness, including the remote login path, which is why nothing short of cutting the power would move it.

Second, the "idle" servers were not idle-sized. The inference framework's internal buffer cache defaulted to an effectively unlimited ceiling and never returned freed allocations to the OS, so a server that had been busy earlier was still holding every scratch buffer from its peak: about 100 GB resident against about 37 GB of weights, and about 64 GB against about 15 GB for the smaller service.

Fixes, in order of value: cap the framework's buffer cache and clear it on an interval; a memory guard that samples free pages and terminates the newest model server before exhaustion; and a preflight check in the launcher that compares live free memory against the declared budget of the model being started, refusing rather than proceeding. The number that matters is resident size, not weight size — they are not close.

If you are stacking AI workloads onto shared hardware or a shared budget, and your sizing lives in a spreadsheet that nobody has checked against live usage, that is a consulting day with a hard number at the end — for your business in Dubai, the UAE, or wherever the machine lives. I will measure what is actually resident across your stack, work out the real headroom, and leave you with a preflight check and a guard that refuses the workload instead of losing the machine. Book a consulting day or send me an inquiry, and I'll open with one question: what is resident right now.