The Service I Built Myself Was the One Without Limits

I was at my laptop, pushing test traffic at a small new service running on the machine in my office. Nothing dramatic — a search-indexing component, the least important thing on the box. Then the machine stopped answering. Not the service. The machine.

No remote session, no console, no response to anything. It came back only after a hard reboot from the front panel — the recovery step that exists because every gentler one has already failed. On a 512 GB M3 Ultra that serves models all day for my work and for clients, that is not a small event.

A minute earlier I had noticed something strange and said it out loud: how is only 19 GB free if I have 512? That was the whole incident in one sentence, and I did not recognise it.

The systems I bought had limits. The one I wrote had none.

That same morning I had done exactly this work on the production model server: capped how much memory it could hold, capped how many requests it would take at once. I knew the failure mode. I had fixed it hours earlier, on the system that matters.

Then I stood up something new in an afternoon — a small embedding model for search, written quickly with an AI coding assistant, an order of magnitude smaller than anything else on the machine. I never asked it the same question.

Here is what it was allowed to do. One request could carry 8 pieces of text at up to 8,192 tokens each: 65,536 tokens in a single call. The model needs roughly 150 KB of working memory per token, so one maximum-size request reserves about 9.8 GB. The framework underneath gives each request its own thread from a shared pool whose default size is 40. Forty at once is roughly 393 GB, on a machine whose main model already holds 398 GB of weights.

Nobody chose 40. It was a library default, behaving exactly as designed.

The lesson for businesses: mature products ship with cautious defaults because the vendor has already had this outage and paid for it. Software you commission — from an agency, an internal team, or an AI assistant — carries no such history. Its limits are whatever you wrote down.

A missing limit is not a bug anyone finds in testing

The code was not wrong. Every line did what it was meant to do. One request at a time it was flawless, and had been all afternoon. It failed the first time it was asked to do several things at once, which is the only way real systems ever use anything. My stress test did not find a bug; it was the first honest traffic that service had ever seen.

Missing-limit failures are shaped like traffic, not like code. Review will not surface them; unit tests will not; a demo certainly will not. If you are signing off on an internal tool this quarter, the question that earns its keep is not "does it work?" but "how many of these can run at once, and what does that cost?" It is a question almost nobody gets asked.

The thing that broke was not the thing that hurt

The embedding service was the least valuable process on that machine. It is not what the outage cost me.

The production model — unrelated to the bug, untouched by that afternoon's work — was caught in the crossfire. Everything on the box draws from one shared pool of memory. As the new service ate into it, the large model was squeezed out, tried to reload its 398 GB of weights, could not get the memory, died, and tried again. The logs show more than 21 restarts in a loop.

Blast radius is not proportional to importance. A low-stakes tool with no ceiling sits in the same memory as the system your revenue depends on, and it is the low-stakes tool that decides when both stop. On shared infrastructure, whatever was built in an afternoon sets your reliability.

When the machine dies, your evidence goes with it

Reconstructing what happened was harder than fixing it.

The system's own log had a 26-minute window containing zero entries. Not sparse — empty. The logging service is a program like any other; it needs memory and a slot on the processor, and got neither. The machine was too starved to write down its own death.

What rescued the investigation was that some services on that machine write their logs straight to files on disk. Those survived the reboot, and the timeline was rebuilt from them. The second fix had nothing to do with memory: the service had been running in a terminal window, where its output lives only as long as the window does. I moved it to a supervised background service that restarts on failure and keeps its own log on disk.

Your ability to investigate an outage is part of the architecture, not something you acquire afterwards. Check where your systems write theirs — held in memory and lost on restart is common, and you find out on the worst day.

The fix was three numbers, not a rewrite

There was no redesign. Three bounds went into the service itself: a cap on how many requests may run at once, a token budget per batch, and a maximum request size so no single call can ask for the moon. Afterwards, 12 simultaneous requests completed, memory inside the new ceiling.

I deliberately did not lean on the model-serving layer beneath it for protection. Its limits are sensible, but they were sized for a different tenant and have no idea what my service intends to allocate. A service capable of taking down a machine needs its own bounds, in its own code.

The bottom line

Anything you stand up yourself needs explicit resource limits written into it, because nothing else will supply them. Vendor defaults are conservative because the vendor has already lived through this; your afternoon project has lived through nothing.

The unglamorous internal tool is the one that takes the important system down, and it waits until the day real traffic finally arrives.

For the engineers

The service exposed a synchronous request handler on an ASGI framework. A sync handler does not run on the event loop — the framework hands it to a worker threadpool whose default size is 40. All 40 threads can be inside the model at once, and nothing in the handler knew about the other 39.

Per-request worst case: 8 texts at 8,192 tokens is 65,536 tokens; at roughly 150 KB of transient allocation per token, that is about 9.8 GB in flight. Multiply by the pool: about 393 GB of transient demand on a box where the resident language model already holds 398 to 399 GB. Free memory was down to 19 GB shortly before the lockup.

Past that point the machine does not degrade, it stops. Unified memory means the GPU allocator and the kernel's own working set draw from one pool, so the pressure starved the system logging daemon — hence the 26-minute hole in the system log where the incident belongs, and file-backed service logs as the only usable evidence. The resident model was evicted, tried to reload its weights, failed, and restart-looped more than 21 times.

The fix: a semaphore around the encode path, a token budget per batch, and a request size limit, all inside the handler, plus the process moved under a supervisor with logs on disk. Twelve concurrent requests then completed with peak memory inside the bound. Bound your own concurrency — the framework default is sized for handlers that allocate kilobytes.

The machine that froze was the same one whose cold-start behaviour I audited in The Reboot That Locked Itself Behind a Login Screen, and the fix here follows the same philosophy as Your AI Agent Shouldn't Have Your Passwords: The Charter Model — every component gets explicit, written-down boundaries rather than inherited trust.

If you have services in production that nobody has ever given a concurrency limit — internal tools, an AI feature built quickly, anything sharing a machine or a database with the system you depend on — start with a consulting day: I will find the ones with no ceiling, tell you what each can consume at full tilt, and leave you with limits written down and logs that outlive the crash. Book a consulting day or send me an inquiry and name the service you trust least.