Silent Failures in the Plumbing

I asked a model to write one large file. It did the work correctly, and I paid for it three times.

Nothing failed the way software normally fails. No crash, no wrong answer, no complaint from the model. What I saw a long pause, then a generic message about the connection being interrupted. What had actually happened was that the same eleven-minute job had been run again from scratch, twice, and nobody thought to mention it.

The file came to roughly 15,700 tokens, produced at about 24.5 tokens per second — eleven minutes of continuous work. The layer sitting between the model and my editor gives up after 600 seconds of silence. The job was a minute short of the line.

A healthy long job and a dead one look identical from the outside

Here is what makes this a plumbing problem rather than a model problem. While the model composes one single large answer, it sends nothing over the network. The output is held internally until fully assembled, then released in one piece. For eleven minutes, the connection carried zero traffic.

The component watching that connection exists to notice when something has died. It has exactly one signal to work with: how long since anything arrived. Ten minutes of nothing is a reasonable definition of dead — until the day a legitimate job takes eleven.

The lesson for businesses: every timeout you own is a bet about how long honest work takes. That bet was placed by someone who did not know what your longest job would eventually look like, and it is settled silently, in your system's favour, right up until it isn't. It is the same class of quiet drift I wrote about in Automation That Survives Contact With Reality.

The retry is where the money goes

Nothing here was badly designed. The connection broke, so the client did the obvious thing and started again. Then it broke again, and it started again. Three runs of an eleven-minute job to get one file.

But the work behind it was neither cheap to repeat nor safe to resume. No checkpoint, no partial file, nothing to pick up from. Every retry began at the first word and paid the full bill.

Now notice which jobs get retried. The timeout fires on length. The longest, most expensive request is by definition the one most likely to trip a silence limit, and therefore the one most likely to be run again. The mechanism selects for exactly the work you could least afford to repeat.

For a business, that is the difference between a retry policy that quietly absorbs a network blip and one that triples your worst invoice on your worst day. Automatic retry is a discount on cheap work and a tax on expensive work, and almost nobody checks which kind theirs is.

Do not raise the timeout without giving the work a voice

The obvious fix is to make the limit bigger. On its own that is a bad fix: it does not restore the missing distinction between working and dead, it just makes a genuinely dead job hang for an hour instead of ten minutes.

So the first change was to make the work announce itself. Both model servers now emit a signal every 15 seconds while a long answer is being composed — nothing meaningful, just a pulse that says still here. Only then did the limits move: the watching layer went from 600 seconds to 3,600, and the client's own limit was raised to match.

I will be honest about what that buys. It stops the false verdict and the duplicated work. It does not make progress visible — while a long call runs, I still see nothing on screen. That is a smaller fix than it sounds, and it is the right one: the expensive failure was never the waiting. It was the silent repetition.

The error names whichever component noticed the damage first

The second failure is older and stranger. Earlier in the year I asked the coding assistant to run two pieces of work at the same time, in parallel. It came back with a cryptic complaint that required parameters were missing, and then looped, retrying, failing the same way.

Every instinct says the assistant is misbehaving, or the model is producing malformed instructions. It was neither. The damage happened in the translation layer between them — the unglamorous code whose only job is to convert what the model says into the shape the assistant expects.

That layer had been written when a response only ever contained one instruction at a time. When two arrived together in the same chunk, it took the first and silently dropped the rest. The second task's instructions never reached the assistant, which then reported, accurately, that a required parameter was missing.

Before the fix, a response carrying two parallel calls produced 8 events with only one call's data present. After it, 11 events, both calls complete and intact. I patched it locally for immediate relief and then sent the same fix upstream to the project's maintainers.

Concurrency does not introduce the bug, it reveals it

That code had been correct for as long as work arrived one item at a time. It did not become wrong when I added parallelism; it had always been wrong, and parallelism was simply the first thing to ask it the question it could not answer. Two agents dispatched together produced that case on every single request.

If you are adding concurrency to a working process — more workers, batched jobs, parallel approvals — expect the breakage in the old parts, not the new. Something in the existing chain is quietly assuming things arrive in single file, and it will not tell you so. It will hand you an error with somebody else's name on it. That is why every pipeline I hand over gets a health-check pass first — Shipped Is Not Running is the general case.

The bottom line

Neither of these was an AI failure. Both were plumbing: a watchdog that could not tell patience from death, and a translator that had only ever been asked to carry one thing.

A retry is a safety feature only when the work behind it is cheap to redo or safe to resume. Otherwise it is a quiet multiplier on your most expensive jobs. Go and find out which of your automations retry themselves, and what one retry actually costs. Almost nobody has that figure to hand.

For the engineers

Both local model servers buffer a tool call in full and emit it only once parsing completes, so a single long generation puts zero bytes on the wire for its entire duration. Roughly 15,700 tokens at about 24.5 tok/s is eleven minutes of that. The proxy in front held a fixed 600-second read timeout, so it severed a healthy stream roughly a minute before completion, and the client re-issued the identical request — twice. Three full generations, one file.

The fix has three parts. Emit a keepalive every 15 seconds from both servers during tool-call generation; raise the proxy read timeout from 600 to 3,600 seconds; raise the client's own request timeout to match. The keepalive is the load-bearing change. Without it, a longer timeout only delays the same wrong verdict. Note what is still unsolved: the client renders nothing until the call returns, so the operator has liveness but no progress.

The earlier defect was in the streaming adapter translating the server's native format into the assistant's expected schema. It assumed one tool call per response and took only the first when several arrived in a single chunk. Two parallel dispatches hit that path on every request. Before: 8 emitted events, one call's arguments present. After: 11 events, both complete. Patched locally, then sent upstream — review there caught a dead loop, a shared-reference bug and a missing test before it was fit to merge.

If you are running agents or pipelines that retry on their own, and nobody has worked out what a retry costs on your longest-running jobs, that is a consulting day worth booking — whether your stack sits in a Dubai office or is spread across your business. I will map where your timeouts sit, what your system actually does when one fires, and whether the work behind them can resume or has to start again from nothing — then leave you with retries that are cheap by design rather than by luck. Book a consulting day or send me an inquiry and tell me the longest-running job you have.