A Test Is Not a Load

I broke a production server by asking a question. Not a big question — I wanted to know whether a larger embedding model would give me better retrieval quality than the one I was already using, and the only honest way to find out was to index a real corpus and compare. Roughly 2,100 chunks of text. Fifteen minutes of work, maybe.

The corpus was small. The server was not idle.

That server — a shared box I run, carrying always-on production workloads including live call handling (the voice agent from I Built an AI Voice Agent That Makes Real Phone Calls) and search infrastructure — went from a median response time of 0.14 seconds to over 3.5 seconds. Twenty-five times slower. Real business processes, the kind where somebody on the other end of a phone line is waiting, got slower because I was curious about a model.

Here is the part that should worry you: the indexing job was not aggressive. It was not parallel. It queued requests one at a time, the most polite pattern I could have written, and that alone was enough. A single-threaded loop against a large embedding model saturated enough of the machine that everything sharing it degraded. I had assumed the danger threshold was somewhere up around concurrency and batching. The danger threshold was "started at all."

how I caught it — which is to say, badly

There was no alert. There was no dashboard turning red. What happened was that I noticed things felt slow, in the vague way you notice a room has gotten cold, and went looking. Then I killed the job.

That is not a monitoring story. That is a luck story. If I had stepped away for coffee, or been in a meeting, or simply been less twitchy about latency that day, the degradation would have run for the full duration of the indexing job and I would have found out from someone else — probably from a customer complaint routed through a client, which is the most expensive possible way to learn that your server is slow.

The second decision mattered more than the first. After killing it, I refused to let the job re-run until I had a full request-by-request audit of what it had actually been doing. Not a summary. Not "it indexes documents." Every call, in order, with timing. Because the whole reason I got surprised is that I had a mental model of the job that was wrong, and re-running it with the same wrong model would just have produced the same outage with more confidence attached.

The audit is the boring part. The audit is also the only part that changes anything.

what the rebuild actually looked like

The retry was not the same job with a shrug and better timing. It was structurally different, and the mechanisms are worth naming because they generalise to any bulk AI job:

Smaller batches. Instead of pushing the corpus through as one continuous stream, the work was cut into small groups. Each group is a natural stopping point. Nothing runs longer than the time it takes to notice it running.

Deliberate pauses. Between batches, the job sleeps. Explicitly. This feels wasteful — you are paying for a server to do nothing — and it is the single cheapest insurance available, because the pause is when everything else on the machine gets to breathe.

A latency-based auto-abort. The job measures how the box is responding and stops itself if response times cross a threshold. Not a warning. A stop.

Checkpointing. An aborted run resumes from where it stopped rather than starting over. Without this, the abort guard is a punishment, and anything that punishes you for stopping will eventually be disabled by you, at 2am, because you want the job to finish.

Those four together turned a 15-minute reckless job into a 40-minute safe one. I would take that trade every time.

the framework — five guardrails before any exploratory job touches shared infrastructure

I wrote this down as a standing rule, and I now apply it in every engagement. "Test it" authorizes a probe, not a load. Someone saying "sure, try it out" has given you permission to poke. They have not given you permission to run 2,100 requests through their production machine. Those are different asks and they need different answers.

1. A stated go-ahead for bulk work, separate from the go-ahead for testing. If the job will make more than a handful of calls, it is a load, and a load needs its own yes. Ask again. Ask specifically: "this will run about 2,000 requests over roughly 40 minutes against the shared server — is now a good time?" Nine times in ten the answer is "not now, do it tonight," which is exactly the answer you wanted and would never have gotten by not asking.

2. A headroom check before you start. What else is on this machine, what is its current load, and what is the busiest hour of the day for the business processes sharing it? Two minutes of looking. Most bulk-job incidents are not sophisticated failures — they are jobs launched at 11am on a Tuesday.

3. Small requests, always. Batch size is a risk dial, not a performance dial. Halving the batch roughly halves the size of the blast when you are wrong, and you will be wrong about how heavy the job is, because I was, with a single-threaded loop over 2,100 chunks.

4. An abort guard. The job watches the health of the thing it is running against and stops itself. This is the one I want you to remember, and I will make the strong version of the claim: an abort guard is worth more than a monitoring dashboard. A dashboard requires a human to be watching it, awake, in front of it, at the moment the graph turns. An abort guard requires nothing. It works during your meeting, during your flight, at 3am on a Friday. Dashboards tell you that you already have a problem. Abort guards mean you had one for eleven seconds.

5. Checkpoints, so aborting is cheap. A safety mechanism that costs you an hour of redone work is a safety mechanism you will route around. Make stopping the easy choice and it stays enabled.

Two related things from the same period, both cheap lessons that cost me nothing only because they landed near the expensive one. During another long-running job, I checked in on progress — just a status poll — and the poll itself measurably degraded disk throughput on the machine. Even watching has a cost. And separately, while tuning worker counts on a different pipeline, I found the real concurrency ceiling was not the API quota everybody had assumed was the limit. It was an unrelated serialized database write, buried three layers down, quietly setting the true speed of the system. Everyone had been optimising against the wrong constraint for weeks. A workload that looks trivial turning out to be the one that eats the machine is the same lesson as One Workload Too Many.

why this bites harder here

In the Gulf, and specifically in the lean operations I mostly work with, this class of failure is not an edge case. It is the default architecture. A twelve-person company running real revenue does not have a staging environment, a dedicated experimentation cluster, and a platform team keeping them apart. They have one box. Sometimes two. The AI experiment, the call system, the search index and the nightly reports all share the same CPU because that is what is affordable and, honestly, because for most of the year it works fine.

That works right up until someone says "let's just try it."

And AI work is unusually good at producing this failure, because the instructions sound so small. "Index my documents." "Summarise last quarter's tickets." "Test the bigger model." Four words that expand into thousands of sustained requests, with no natural signal in the phrasing that anything heavy is about to happen. A developer asking to run a migration knows they are asking for something. Someone asking to try an embedding model does not feel like they are asking for anything at all.

The fix is not more capacity. More capacity just raises the ceiling you eventually hit while running something bigger. The fix is that the experiment yields to the business instead of competing with it — automatically, without a human in the loop, every time.

where I land

I now run all five of these guardrails as standard in my engagements, and the abort guard is non-negotiable: no bulk AI job I build touches shared infrastructure without one. It has cost me perhaps thirty extra minutes of engineering per pipeline. It has saved at least one client afternoon that I know about, and probably several I never noticed, which is rather the point.

If you are running AI experiments on the same machine that runs your business — and if you are a lean team, you almost certainly are — the useful exercise is not buying a bigger server. It is spending an hour finding out what your current jobs would do to it, and adding the guard that makes them stop. I broke my own production box learning this, so on a consulting day I do that hour with you first, before anything else gets run. I'm happy to walk through it — bring your ugliest script. Book a consulting day or send me an inquiry first if you'd rather talk before booking.