Shipped Is Not Running

Three times in a single day, on an internal tool I run, I was told a fix was shipped and tested. Three times the person who owns the tool opened it and said, plainly, "I don't see it." The repository was correct every time. The tests passed every time. The screen in front of the owner was showing code from an hour and a half earlier, because the server process that was serving that page had started before the commit existed and nobody had restarted it.

That is the whole failure. It sounds trivial written down. It cost most of a working day.

I went back through one month of my own logs and counted the same shape of failure at least eight separate times, across different systems, different tools, different causes. Not eight bugs. Eight instances of one specific lie: shipped is not running. The code was written correctly, committed correctly, and had no effect on the world.

what happened — a month of the same failure wearing different hats

The stale server process was the most common version, but it was far from the only one. A python edit that was supposed to patch a config file never ran at all — the command was chained behind a directory change that failed, so the whole chain short-circuited silently and the agent driving it reported success. Nothing happened. The report said otherwise.

A browser service worker on a small internal app served three-hour-old JavaScript to the client. Every fix "shipped" in that window was, from the user's point of view, pure fiction. The fix was in the bundle. The bundle was on the server. The browser was serving a cached copy from before lunch and would happily have kept doing so for days.

Then the quieter ones. Config edits to a model gateway that were completely silent until a restart — the running server kept advertising an old model list, and one of the configured default models had been retired on the backend entirely. Nobody found out until a routine restart weeks later. A scheduled job that ignored a same-day config fix because the operating system's job manager only re-reads job definitions on an explicit reload, never on a file edit. A background listener holding stale code because a restart had killed an in-flight child worker mid-flight.

Seven mechanisms. One category. In every case the git history was immaculate.

the fix — stop trusting the process that is making the claim

Here is the part that matters, and it is not a tool. When something reported success, I stopped accepting the report and started asking for two independent pieces of runtime evidence.

First: compare the live process's start time to the commit timestamp. If the process started before the commit landed, the running code cannot possibly contain the change — that is arithmetic, not opinion. One of those three incidents was actually proved correct this way: the commit landed at 22:32:37 and the process started at 22:32:38. One second. That is the difference between "we have a caching bug" and "we have a deployment problem," and it took ten seconds to settle.

Second: confirm the served bytes contain the change. Not the file on disk. Not the build output. Fetch the actual asset the actual user's browser receives and look for the string that should now be there. If the served bytes don't have it, nothing else is relevant.

For the service worker, the mechanism fix was rewriting the caching strategy to network-first, so the client asks the server before falling back to cache rather than the other way round. For the scheduler it was adding an explicit reload to the change procedure. For the model gateway it was a restart-and-verify step that queries the live endpoint for its model list rather than reading the config file back.

Different fixes. One principle: never accept a claim about the world that is made from inside the process making the claim. A coding agent inspecting its own work is inside the box. It can tell you what it wrote. It genuinely cannot tell you what is running, and it will state success with total confidence anyway, because from where it sits the work is finished.

definition of done — the list to hand your developer

You don't need to understand any of the above to use this. Hand these five items to any developer, agency, or agent as your standing definition of "done." Refuse to close a ticket without them.

1. Show me the running process, not the commit. What time did the live server process start? What time did the change get committed? If the process is older than the change, it is not deployed. This is a one-line answer.
2. Show me the response, not the repository. Fetch the page or the API the real user hits, and point to the new behaviour in what came back. The file on disk is not evidence. The bytes on the wire are.
3. Tell me what step makes it live, and confirm that step ran. Every system has one: a restart, a reload, a cache purge, a deploy command. Name it. Then prove it executed, and prove it didn't fail silently in the middle of a chain.
4. Verify from a cold client. A hard refresh, a private window, or a different device. This is what caught the three-hour-old JavaScript. Your own browser is the most compromised witness in the building.
5. Have the owner confirm from their own screen before the ticket closes. Not the developer's screen. Not a screenshot from an hour ago. The person who asked for the change, looking at it, today.

None of that requires technical skill. It requires refusing to accept a self-report as an observation.

why AI-assisted delivery makes this worse, not better

The intuitive story is that AI coding tools reduce this risk, because they're fast, tireless, and don't forget steps. The opposite is true, and the reason is arithmetic rather than anything mystical.

Deployment verification is the one step that has not been automated. Code generation went up perhaps five to ten times in volume. Review is partially automated. Testing is partially automated. But confirming that a change is genuinely live in front of a genuine user is still a manual act performed by a human being with a browser. So you have five to ten times more shipped changes flowing through a verification step whose capacity has not moved at all.

That gap doesn't stay empty. It fills with confident, well-written, entirely fictional progress reports. And agents are unusually good at producing those, because they write clean summaries of what they intended to do and have no independent channel to check whether it happened — the same self-reporting failure I ran into asking an agent to verify its own work overnight. My month of eight failures was not a month of bad tooling. It was a month of good tooling outrunning the last manual checkpoint.

Faster. Cleaner. Wrong.

why this bites harder here

In the Gulf, a very large share of the businesses I work with run genuinely lean technical teams — often one internal person, often nobody, with an outside agency or a contractor holding the actual keys. There is no second engineer who happens to open the app and notice it looks the same as yesterday. There is no QA function. The feedback loop between "the developer says it's done" and "someone discovers it isn't" can run for weeks.

That gap is exactly where budget disappears. You pay for work that was genuinely performed, genuinely committed, and genuinely never reached a user — and because the code is real, nobody involved is lying, so nobody gets caught. Add the regional norm of remote or offshore delivery and the owner rarely sees the system running in a context they control. The invoice is concrete. The deployment is not.

Lean teams don't need more process. They need one checkpoint that cannot be faked from inside the system doing the work.

close

I now run that five-point definition of done as standard in every engagement, and I apply it to my own agents before I apply it to anyone else's. It has caught more real problems than any code review I've done this year, and it takes about ninety seconds per change. The same discipline applied to the signals your systems report is what I describe in Every Proxy Signal Lies Eventually.

If you're paying for development work — agency, contractor, or an internal team leaning hard on AI tooling — and you have a nagging sense that the velocity you're being told about doesn't match what you actually see in the product, that's usually this. Not incompetence. A missing checkpoint. I lived this on my own tooling before I ever taught it, and on a consulting day I walk your delivery process end to end and install exactly this definition of done, tuned to how your business actually ships. If you're in Dubai or across the UAE, book a consulting day or send me an inquiry first if you'd rather talk before booking.