Every Proxy Signal Lies Eventually

A job in an operations system I run finished with exit code 0. Clean success. The model it had called returned a completely empty response — zero characters, no content, nothing — and the wrapper around it shrugged, exited zero, and moved on. No retry. No alert. Eight hours of downstream work built on an empty string.

Exit code 0 means a process ended without crashing. It does not mean work happened. I knew that in the abstract. I had still built a pipeline that treated the two as identical, because checking the exit code took one line and checking the output took thought.

That is the whole story of this post. Every convenient signal you use to check whether something worked is a proxy for the real thing. And proxies drift.

what happened — the catalogue

Once I started looking, I found eleven of them in about six weeks across my own systems and two client engagements. Not exotic bugs. Ordinary, boring, everywhere.

A monitor reported a long-running task as "still in progress" because the watcher process was alive. The underlying call had never connected — not once, not partially. The watcher itself was hung, waiting on a socket that was never going to answer. Process alive is not operation succeeding. It is not even operation started.

A build session showed as running for eight minutes. It was dead. Its process had been gone the whole time. I only found out by opening its raw transcript file and reading the last line by hand. Another session in the same batch died on a rate-limit error with zero commits written and zero error reported — from the outside, that is pixel-for-pixel identical to a session working quietly on something hard.

An API returned 200 OK for marking a batch of messages read. It was a silent no-op. The server had declined the work because of an identity-format mismatch in the request, and then told me everything was fine. A success response for work that was never performed. That one took three days to notice because the symptom was elsewhere: the same messages kept coming back.

Then the inversions — where the check screamed and nothing was wrong.

A verification script reported that all databases were corrupted. Every single one. They were perfectly healthy; the script had a bug in how it constructed its own file paths and was checking files that did not exist. The tell was the uniformity. Real corruption is messy — three bad, nine fine, one weird. When failure is perfectly uniform, suspect the checker, not the fleet.

A monitoring alert warned that a required program was "not on PATH." The monitor was inspecting its own environment, not the worker's. Purely cosmetic. Also indistinguishable, at 2am on a phone screen, from a genuine production outage.

A resource monitor showed 0% GPU usage for a workload that was unambiguously GPU-bound — the fans were audible, the job was 40x faster than CPU. The tool was reading the graphics counter instead of the compute counter. Two different numbers on the same chip.

A physical dial on a storage enclosure appeared to be set correctly. The setting had never committed to the controller. I confirmed the real state only by pulling the actual disk inventory and counting.

A status report said 8,000 files remained to process. It had been read from the wrong database column. The real backlog was roughly four times larger and a materially different category of work. And underneath all of it, failures were being recorded as though an escalation step had never been needed — quietly masking real content loss as routine success.

Eleven signals. Eleven green lights. Zero of them lying on purpose.

the fix — why cheap signals fail

Here is the pattern, and it is not a coincidence. Every one of those signals was chosen because it was cheap to check.

Exit code: one integer, already there. Process alive: one syscall. HTTP status: already in the response object. GPU percent: one number in a dashboard. Row count: one query. Nobody sat down and chose a bad signal. They chose the available signal, which is always the one sitting closest to hand.

But cheapness and correctness come from the same place. A signal is cheap precisely because it is downstream of or adjacent to the thing you care about rather than being the thing itself. Exit code is downstream of the process, not the output. Process-alive is adjacent to progress. HTTP 200 is the transport layer's opinion about a request, not the application layer's report on the work. The gap is small at first. It is always small at first. Then something changes on one side — a model starts returning empty responses under load, a server tightens its identity parsing — and the two definitions separate, silently, with the light still green.

The fix in my systems was not a better monitoring tool. It was writing down, for each check I relied on, the exact sentence describing the gap. "Exit code 0 means the wrapper did not crash. It says nothing about whether the response body was non-empty." Once that sentence exists, the missing assertion is obvious. I added a length check. Four lines.

Do that for every check and the honest ones survive and the frauds fall out fast. In my case, three of eleven checks were load-bearing and correct. The rest were decoration.

the framework — five questions before you trust a check

Run these on any automated system, yours or a vendor's. Monday morning, no tooling required.

1. What does this check literally measure — in one sentence, no interpretation? Not "it checks the job succeeded." Write the mechanical truth: "it reads the exit status of a shell process." If you can't write that sentence, you don't know what your dashboard says.

2. What is the precise gap between that and what I believe it means? Every check has a gap. The question is never whether one exists, only whether you've named it. An unnamed gap is where the outage lives.

3. Am I verifying the outcome or the mechanism? This is the load-bearing one. Check that the message arrived, not that send() was called. Check that the file exists at the destination with the expected byte count, not that the upload function returned. Check that the row is in the table, not that the insert didn't throw. Mechanism checks tell you your code ran. Outcome checks tell you the world changed. Only one of those is what you're being paid for.

4. What does silence mean here — success, or nothing at all? A dead session and a working session look identical if your only signal is absence of complaint. Any system where "no news" is your success indicator needs a positive heartbeat that carries evidence of work: a count, a timestamp, an artifact. Not a ping.

5. If this check reported something surprising, would I suspect the system or the check? Correct answer: the check. First. Always. A monitoring system is a system, built by the same people under the same time pressure, and it breaks in exactly the same ways. Uniform failure, impossible numbers, everything-is-broken alerts — those are check bugs until proven otherwise.

why this bites harder here

In the Gulf I mostly work with lean teams. A 12-person company running operations that would carry 40 people elsewhere, a family business where one person owns three systems, a fast-growing firm where the person who built the pipeline has since moved to a different department or a different country entirely.

Lean teams cannot afford redundant verification, so they lean harder on automation — which is the right call, and it makes proxy signals more dangerous, not less. When one person watches nine dashboards, they are not investigating amber. They are scanning for red. A green light that means nothing is functionally invisible to them, and the failure surfaces weeks later as a customer complaint or a compliance gap rather than as an alert. The complementary failure — expected events that never happen and nobody notices — is the subject of The Failure Path.

There is a second thing specific to this market. A lot of AI deployments here are recent — eighteen months old, built fast, often by an outside vendor who has since finished the engagement. Nobody who remains at the company has ever read the verification logic. It was inherited, it is green, and green is assumed to mean something. I have opened three such systems in the past year. In all three, the monitoring was checking the mechanism and reporting on the outcome. Nobody had lied. Nobody had checked either.

The cost is never the incident. It is the interval — the weeks between the thing breaking and anyone finding out, during which you keep making decisions on numbers that stopped being true.

close

I now run a signal audit as standard in every engagement — the first week, before I touch anything else. Every automated check in the system gets the five questions above, and each one ends up in one of three buckets: verifies the outcome, verifies the mechanism, or verifies nothing at all. That third bucket is never empty. It is usually the biggest.

It is not glamorous work. It is a spreadsheet and a lot of reading. It also has the best return of anything I do, because you cannot improve a system whose reports you can't trust — every optimization after that is guesswork wearing a number. For teams that want the monitoring to go further, A Test Is Not a Load covers the guard that acts when no human is watching at all.

If you're running AI or automation in production and you're not certain what your green lights actually measure, that audit is a contained piece of work — a few days, and you get a written list of which of your checks are load-bearing and which are decoration. I built the empty-response pipeline that started this post, so I audit my own signals before anyone else's. If that sounds useful for your business in Dubai or across the UAE, book a consulting day or send me an inquiry first if you'd rather talk before booking.