Two Outages, Three Wrong Suspects

Two things broke on the same day, both in the voice assistant I run on the machine in my office. The first stopped accepting connections outright, mid-conversation, while I was using it. The second kept cutting out mid-sentence and survived two correct fixes.

In both cases the obvious suspect was my own server. In both cases the obvious suspect was wrong.

The first fault was a connection that took 60 to 90 seconds to hand back a resource it had never used. The second was a machine on my own network, not part of the audio path at all, that had been resending the same 44,526-token request roughly every eight seconds for about eight hours — including during every clean measurement I took to prove my server was innocent.

Capacity was not the problem. Nothing was giving capacity back.

The speech-to-text service in front of the assistant was configured to allow exactly one conversation at a time. That was deliberate. So when it started refusing connections, there were two explanations available, and they lead to opposite actions: either one channel is genuinely too few, or something is failing to release the channel it holds.

The measurement settled it. A client that connected and then disconnected without ever sending audio held its upstream channel for 60 to 90 seconds before anything freed it. Two reconnection attempts in quick succession were enough to consume everything available — and reconnecting quickly is exactly what a frontend does when the connection is already unhappy. This is one of the ways an AI phone call falls apart that never shows up in a demo — see Six Ways an AI Phone Call Falls Apart.

So the service died from the very behaviour its own failure produced. It worked perfectly in every ordinary test, because ordinary tests send audio.

The lesson for businesses: a service that falls over under retries is rarely short of capacity. It is usually leaking capacity, and the leak stays invisible until something upstream starts retrying — which is to say, on the worst day, at the worst moment.

Raise a limit only after you have proved the room empties

With the leak fixed, releasing a channel took 0.28 seconds through the proxy, and 0.02 seconds when I tested straight against the service behind it. Only then did I raise the concurrency limit from one to 64, matching the default the upstream project ships.

I priced that change before making it, because headroom you cannot cost is a guess. Resident memory stayed flat at 2.14 to 2.15 GB whether the limit was 1, 16 or 64. Single-stream latency was unchanged — 4.40 seconds against 4.41. A test with 24 simultaneous connections passed cleanly. But a burst of 24 polite connections is still not a real load — A Test Is Not a Load is the fuller argument about why.

The order is the whole discipline. Capacity alone would have saved nothing: a leak drains 64 channels as reliably as it drains one, just more slowly. The leak fix alone would have left a service I depend on running with a single channel of margin. Fix the release, then buy the headroom, then confirm the headroom cost nothing.

Two real bugs, correctly fixed, and the symptom stayed

The second outage was the assistant cutting out mid-reply during calls. I found two genuine faults and fixed both, with numbers on either side.

The first was startup: time to first audio came down from 0.68 seconds to 0.42. The second was delivery — the speech engine was handing out audio in growing bursts with silence gaps of up to 1.7 seconds between them. Once it was made to release audio at a steady rate instead, the largest gap between sounds was 0.08 to 0.09 seconds, with no dropouts at all, on replies ranging from 6 to 48 seconds long.

Both fixes were right. The calls got noticeably better. Over a two-minute conversation it still cut three times.

Two correct fixes that improve a symptom are the most expensive thing in an investigation, because they lend the wrong theory credibility. Progress feels like proof, and it isn't.

Every clean test I ran landed in the gaps

The third suspect was a small background model sharing the same processor. I tested it directly: one large request to it, 5,000 tokens, produced no audio dropout. Theory dead. Then I tested it the way a live call actually behaves — nine of those requests at once — and measured a 3.7-second hole in the audio. Theory alive again.

Both results were true. Neither was the answer.

The answer was a second machine on my network, nothing to do with the audio path, stuck resending an identical 44,526-token request roughly every eight seconds, more than 76 times, continuously since the previous afternoon. It had been occupying the shared processor around 60% of the time for eight hours — including during every control measurement I had taken that night to establish a clean baseline. Those measurements had simply landed, by luck, in the pauses between its retry cycles.

A week of engineering or an afternoon — the difference comes down to this: a controlled test that does not reproduce the concurrency of real traffic can confirm a false theory just as convincingly as a true one. And a baseline is only a baseline if you know what else was running while you measured it. The longer version of that trap — trusting a live dashboard over the incident history — is in Can You Promise It Never Happens Again?.

Some fixes do not belong on your side of the wire

Once the loop was found, I stopped tuning the audio server. There were more settings I could have adjusted, and none of them were the fix. The fix was stopping the retry loop on the other machine, which meant writing it up for the engineering team that owns it rather than quietly absorbing the problem into my own code.

That restraint is worth naming, because the alternative is so tempting. Every symptom you compensate for locally makes your component permanently more complicated than its job requires — and leaves the real fault in place, free to come back somewhere you are not watching.

The bottom line

Before you accept that your own system is still broken, find out what else is using the same machine. A shared resource lets an unrelated fault somewhere else look exactly like your code misbehaving at random, and a leak lets a service exhaust capacity it was never actually using.

Neither failure appears in a happy-path test. Both appear the moment something starts retrying.

For the engineers

The proxy ran two concurrent tasks per connection: one pulling audio in from the client, one relaying transcription results back out. When a client disconnected without ever sending audio, the outbound task had nothing to push, never reached the error condition that triggers cleanup, and blocked indefinitely holding its upstream channel. The ordinary case self-healed — audio had been sent, a later push to the now-closed client socket raised, cleanup ran — which is precisely why the bug was invisible under normal use and only surfaced under the rapid, audio-free reconnects a struggling client produces. Post-fix release time: 0.28 seconds through the proxy, 0.02 direct. Raising the channel limit from 1 to 64 left resident memory flat at 2.14 to 2.15 GB and single-stream latency at 4.40 seconds against 4.41; 24 concurrent connections passed.

The audio pipeline had two separate defects: cold-start latency, 0.68 seconds to first audio down to 0.42; and a decode-and-emit pattern that fell behind real time as replies grew, producing bursts with inter-frame gaps up to 1.7 seconds. Pacing emission to a steady clock and bounding the decode window gave a verified maximum gap of 0.08 to 0.09 seconds and zero underruns across replies of 6 to 48 seconds. The residual dropouts were external: a single 5,000-token request to the small resident Gemma model caused no underrun, nine concurrent ones caused a 3.7-second gap, and the true load was a client outside the pipeline entirely, resending 44,526 tokens every eight seconds for eight hours.

If your systems fail intermittently — a service that collapses under reconnects, calls that drop for reasons nobody can reproduce, an incident where two correct fixes still left the symptom alive — that is a consulting day I have run many times, including for voice AI systems across the UAE. I will map what actually shares a machine with what, find the resources nothing is handing back, and leave you with tests that reproduce the concurrency of real traffic instead of one tidy request at a time. Book a consulting day or send me an inquiry with the symptom nobody on your team can reproduce.