I Broke My Own Safety Rule During a Live Call

I was on a live voice call with my own assistant, listening for an audio glitch I had spent most of the day chasing. Mid-sentence the other end went quiet. Not the assistant — the machine it runs on. No remote session, no console, no login prompt. The only reason it came back at all is that I was sitting in the same room.

While I was on that call, an AI coding assistant working alongside me had been benchmarking the same machine: five restarts of the main model server, then a second full copy of a model loaded to see how two processes compete for the GPU. Roughly ten minutes of sustained, undisclosed load on a 512 GB M3 Ultra that was, at that moment, carrying my phone call.

That morning I had written a safety brief for that machine. It said in plain words that sustained load tests do not run without telling me first. I wrote the rule and broke it inside the same working day.

A rule nothing enforces is only a preference

The brief was not vague. It named two specific hazards. One: an automatic keep-alive policy will cheerfully resurrect a broken service and report it as healthy, so restarting is not proof of anything. Two: no sustained load testing without prior notice. The incident violated both — while debugging the exact problem the rule existed to keep uncomplicated.

This is the uncomfortable part. The document was correct, current, and written hours earlier by the person who owns the machine, with this exact failure in mind. None of that mattered, because nothing checked whether anyone was following it.

The lesson for businesses: a written operating rule delivers value only through adherence, and adherence is not a property of the document. It is a property of the machinery around it — an approval step, a maintenance window, a check that refuses the action. I have none of those yet. I have a document and my own attention, and that afternoon my attention was on a phone call. That is the same limit behind most of the policies in your shared drive: if a rule can be broken by a competent, well-intentioned person who is busy, it will be.

A reboot is an audit you did not schedule

Coming back up, the machine handed me two failures I did not know it had.

The database service that one of my components depends on refused to start. And separately, callers had been hearing the wrong voice — the old, slower text-to-speech engine — for about an hour, with nobody noticing.

Neither was caused by the crash. Both had been sitting there quietly. The crash simply forced the machine to do the one thing that reveals them: start from nothing.

For a business, that is the real argument for rehearsing a cold start. If you cannot remember the last time your production stack came up from zero, you do not know whether it can. Every unclean shutdown and every leftover from a migration is stored in the boot path, and it all comes due on the same day — usually the day you are already having.

"Keep the old one for rollback" is a race condition with a nicer name

When I swapped the text-to-speech engine, I left the old service installed and enabled, for rollback. Both definitions claimed the same network port. Both were set to restart automatically. On every boot, whichever one got there first owned the port and became the voice that callers actually heard.

Nothing reported an error. From each service's point of view the situation was normal — one serving, one retrying. For roughly an hour real calls went out on the old engine, and I only found out because I happened to be rebooting for an unrelated reason.

When you retire a production service, remove it. A rollback plan is a documented procedure and a stored build you can reinstall — not a second live copy of the old system with automatic restart switched on. And whatever you swap, add one check that answers a different question from the usual monitoring: not "is something running?" but "which version is answering right now?" Almost nobody monitors that, which is why a silent downgrade can run for an hour.

Stop letting machines identify things by description

The database refused to start because of a stale lock file. The file said, in effect: I am already running, and here is my process number. The number was real. There genuinely was a live process with that number. It was not the database — on reboot the operating system had handed that number to something else entirely, a background model server that happened to start earlier.

A record described a thing. The machine trusted the description, matched the wrong object, and drew a confident, wrong conclusion.

The same shape shows up in my own automation. A long benchmark chain kept dying partway through with exit code 144, twice, taking hours of queued work with it. Its cleanup step found the model server to shut down by searching the running process list for text from the server's launch command. That text also appeared in the command line of the wrapper script running the whole chain. "Stop the server" matched the supervisor and killed it mid-run, discarding everything still queued and the partial results already collected.

Both are the same mistake at different altitudes: identifying something by what it looks like rather than by what it is. If your automation kills, restarts or cleans up anything by matching a name, it will one day match something you did not mean — and the failure will look like a mystery rather than a bug.

The bottom line

The rule was right, current, and written by me that morning. It still did not survive contact with a busy afternoon. Written rules are not controls; they are intentions with better formatting, and the gap between the two is where outages live.

And when the machine finally came back up, it did not just recover — it published a list of everything I had left half-finished. Reboots are honest that way.

For the engineers

The lock was a PID file left by an unclean shutdown. The startup path read the recorded PID, found a live process holding that ID, concluded the database was already up, and refused to clear the lock or start. The PID was genuine and belonged to something else: after a cold boot, process IDs are handed out from the beginning again, and a background model server had been given that number. The startup script now clears a lock only when three independent checks agree it is stale — no process with that PID exists, or a process exists but is not the database, and nothing is currently listening on the database's port. Any one of those on its own is a coin flip.

The port collision was two service definitions for the same port, both enabled with automatic restart, left behind by a text-to-speech swap where the old engine was kept for rollback. Boot order decided which bound the socket; the loser retried and lost. No health check compared the engine actually answering against the engine intended, so an hour of calls went out on the old voice. The affected service was rolled back to its last known-good build.

The benchmark chain was the same bug one level up: cleanup killed by pattern-matching the process table against the server's launch string, which also appeared in the supervising shell's own command line. Capture the PID at spawn, kill that exact PID, then verify it is gone.

The principle that written rules need enforcement machinery is the same one behind Your AI Agent Shouldn't Have Your Passwords: The Charter Model — a charter that only works because a system checks it, not because everyone remembers it. And the reboot that exposed all of this is the same cold-start test I argue for in Automation That Survives Contact With Reality.

If you are running production systems where the safety rules live in a document nobody can enforce, where retired services are still installed "just in case," or where scripts stop things by matching names in a process list, that is a good use of an on-site day: I will sit with your stack, force a cold start, and show you what your boot path is actually holding — then leave you with checks that identify things by identity rather than by description. Book a consulting day or send me an inquiry now, while nothing has forced a cold start yet.