Shipping With a Bug You Can Name

By six in the morning the feature worked, and worked well: a request that had taken 24.1 seconds cold came back in 4.5 seconds warm, because the expensive part had been saved to disk instead of thrown away. How it got built overnight is a separate story.

This one starts with the last check before I sent it to the maintainers of the open-source inference server it belongs to. I asked the server to stop politely: finish what you're writing, close up, exit. It gets 30 seconds to do that.

It never used them. Not once, in any reproduction. Every single time I had to kill the process outright — the computing equivalent of pulling the plug out of the wall. And then I sent the feature anyway, with that failure written into the first part of the note that went with it.

The bug was never the decision

The decision was what to do about a piece of work that was finished, valuable and flawed at the same time.

There were two honest options. Hold everything back until I understood why the shutdown hung — an hour or a week, because "the process doesn't exit" is either trivial or genuinely hard, and you can't tell which until you're inside it. Or release the useful part now and describe the flaw precisely to the people who would inherit it.

I picked the second. Not because I was in a hurry, but because I could answer one question about the bug, and that answer changed everything.

The lesson for businesses: the presence of a known defect tells you almost nothing. Every system you run today ships with defects. What matters is whether anyone can describe them.

Ask what happens when it dies badly, not whether it dies

The question I could answer was: when this thing is killed mid-shutdown, what is lost?

The feature writes each piece of saved work to a temporary file and only then renames it into place. A rename either happens or it doesn't — there is no half-renamed file. So a hard kill in the middle of a write can leave you with the old good file or the new good file, never a corrupted mixture. Worst case, one piece of in-flight work is dropped and recalculated later, which costs a few seconds.

That's the whole basis for shipping it. The bug is ugly, loud and immediately visible — the process refuses to exit and has to be killed. It is not silent, it does not lose data, and it does not degrade anything downstream.

That is the distinction I'd want every operator to hold onto. A bug that announces itself and destroys nothing is an inconvenience. A bug that stays quiet and corrupts something is an incident with a delay fuse on it. They can look identical on a status page — and companies routinely ship the second kind while blocking releases over the first.

A defect you can describe is cheaper than one you can't

There's a version of engineering discipline that sounds responsible and is actually expensive: nothing ships with known issues.

In practice that policy doesn't produce systems without bugs. It produces systems where bugs stop being mentioned. Naming a defect costs you your release date, so defects get reclassified as quirks, or found later by a customer, or fixed quietly in a way nobody can audit. You haven't reduced your risk. You've moved it somewhere with no paper trail.

Compare the two positions available to me that morning. One: a working feature plus a written statement saying here is exactly what breaks, here is what it cannot damage, here is what I think causes it. Two: the same feature a week later, with the same bug found by somebody who had no idea it was there.

The first is a managed risk. The second is a surprise. And surprises are what actually cost money — not defects.

Write the flaw into the shipping note, not the backlog

The mechanical part of this matters more than the philosophy.

A known bug that lives in a backlog ticket has been recorded, not disclosed. Nobody downstream reads your backlog. When the person who has to operate the thing at 2am hits the failure, "we knew about that" is the worst sentence they can hear.

So the disclosure went where the work went. Symptom: asking it to stop cleanly hangs and it must be force-killed. What still works despite that: nothing is corrupted or lost, because of the rename behaviour. What I think is going on: two specific hypotheses, both testable, neither confirmed. Planned follow-up: a fix, separately.

For a business, this is the difference between a supplier who tells you the limitation on the way in and a supplier who tells you during your outage. You will forgive the first one. You will replace the second — and you'll be right to, because they've shown you their disclosure habit under pressure, and it's the same habit they'll have next time.

The disclosure is the part you can actually check

Most buyers can't evaluate whether a vendor's AI system is well built. Almost nobody can, from the outside, in a procurement cycle. But everyone can evaluate how the vendor talks about what's broken.

So ask directly, and listen to the shape of the answer. What's broken in this system today? What happens to my data when it fails? Which failure modes are noisy, and which are silent?

A team that has done the work will answer in specifics — with a symptom, a boundary, a reason it's safe. A team that hasn't will tell you their system is stable. Stable is not an answer; it's a mood. And if you're buying AI in particular, apply this hardest, because these systems fail in ways that look like normal output rather than like errors.

The bottom line

Shipping with a known bug isn't a compromise when you can name it, bound it and prove it destroys nothing — it's just honest engineering, and it's faster than the alternative. What makes it acceptable is the naming, not the bug. The teams to worry about aren't the ones with defects in their release notes. They're the ones whose release notes are clean.

For the engineers

The feature is an opt-in disk tier behind the server's in-memory prompt cache. On a stop signal, the handler calls the disk cache's shutdown with a 30-second drain timeout so the background writer can finish queued entries and exit. It never returned within the budget — a hard kill was required in every reproduction.

Two hypotheses, neither confirmed. Python delivers signals on the main thread, but if that thread is parked inside a C extension call — a GPU kernel launch, for instance — the handler is queued until the call returns; alternatively the HTTP server's serve-forever loop holds the main thread while the handler runs elsewhere. The probe list starts with shutting the HTTP server down first inside the handler, then confirming the writer thread signals completion before the handler releases its lock.

What made it shippable is the write path. Each entry is serialised to a temp file and moved into place with a POSIX rename, which is atomic — a kill mid-write leaves either the prior entry or the new one, never a torn file. That was verified end to end: after the hard kill, a fresh process rebuilt its index from disk at startup and served 15 of 16 prompt tokens from the persisted cache rather than reprocessing them, for the 24.1s to 4.5s difference. Correctness never depended on the graceful path working. The disclosure said exactly that, and the fix went on the follow-up list instead of the critical path.

The feature in question is the durable cache from The Cache That Vanished Every Restart — that post is about the 5x speedup; this one is about what made it honest to ship. And if you're the buyer rather than the builder, Build vs. Buy vs. Blend is the decision that puts you in front of a vendor's disclosure habits in the first place.

If your team is sitting on a release that's blocked by a defect nobody has actually characterised — or, more likely, shipping quietly around several that nobody has written down — give it a consulting day: I'll go through what you're carrying, separate the loud failures from the silent ones, and leave you with a disclosure practice your customers and your on-call engineers can both live with. Book a consulting day or send me an inquiry first if you'd rather talk before booking.