The Reliability Fix That Was a Security Regression

what happened — four days of "still broken"

For four consecutive days, the same line appeared in my morning status report: still broken. A local transcription service on a machine I run kept becoming unreachable. The machine would go to sleep, or drop off the network, and the automated pipeline that depends on it would sit there and hang for a full two minutes before giving up. Four days. Real work blocked, every one of them.

That is the kind of failure that makes you impatient, and impatience is where bad security decisions come from. Nobody widens a trust boundary on purpose. They widen it because something has been broken for four days and they want it to stop being broken. The ticket in front of me did not say "expand attack surface." It said fix the outage.

I fixed the outage. Then I threw the fix away eight hours later, on purpose. This post is about why.

the fix — and it was a good fix

The mechanism was straightforward and, honestly, the kind of thing I would have approved without blinking if a contractor had handed it to me. Two parts.

First, a three-address fallback chain. Instead of resolving one hostname and dying when that failed, the pipeline would try a primary address, fall back to a secondary one, and finally fall back to a raw address on the local network. Three chances to find the service instead of one. Classic resilience engineering.

Second, a five-second socket pre-probe. Before committing to a full connection attempt, the pipeline would poke the address and see if anything was listening. If nothing answered in five seconds, move on. That single change took the failure path from about two minutes down to roughly nineteen seconds — a 6x improvement in how fast the system gives up and tells you it is unhappy.

It worked. I tested it. It shipped. And the four-day outage stopped being a four-day outage.

Then I reverted the whole thing.

why the good fix was the wrong fix

Here is the inversion, and it took me an embarrassingly long time to see it. A fallback chain is not just a reliability feature. It is a permissions change.

A pipeline that will only ever connect to one address is a pipeline with a very narrow blast radius. If someone poisons DNS, or gets onto the local network, or manages to influence a config file, there is exactly one thing they can redirect. A pipeline that will automatically try three addresses is a pipeline that has been taught a new behaviour: when the expected thing is not there, go looking. Keep going. Try the next one.

That behaviour is the vulnerability. Not any of the three addresses individually — the willingness to hunt. A system that will try three addresses is architecturally a system that can be pointed at a fourth, because the hard part was never adding an address to a list. The hard part was convincing the system that trying alternative addresses is normal. Once that is normal, the list is just configuration, and configuration is the softest thing in any deployment.

And falling back to a raw local network address made it worse in a specific way: it took a resolution path that a human had to deliberately set up and replaced it with one that answers to whoever happens to be sitting at that address on that network. On a home or small-office network, that is a much larger cast of characters than anyone imagines.

The revert itself was not free either, which is its own lesson. Pulling the fallback chain out required manually repairing a stale config file that had been written in the new format and would have quietly broken the restored single-host code. Twenty minutes of careful work to undo eight hours of good work. I logged a note to myself afterward, and it is the most useful thing I got out of the whole episode: raise attack-surface concerns before building network-address changes, not after. Review is cheap at the design stage. At the revert stage you pay for the build, the revert, and the cleanup.

the framework — five questions before you approve a resilience change

I now run this checklist on every change that makes a system "more reliable." It takes about ten minutes and it has caught two things this year already.

1. Does this change the set of things the system will talk to without a human present? If yes, this is an access change wearing a reliability costume. Route it to whoever reviews access changes. Retry logic against the same endpoint is fine. New endpoints are not the same category, no matter how the ticket is worded.

2. Where does the list of fallbacks live, and who can write to it? A hardcoded list in reviewed source is one risk profile. A list in a config file, an environment variable, or a database row is a completely different one. Most fallback chains start hardcoded and drift into configuration within a quarter, because that is what makes them convenient. Assume the drift.

3. Is any fallback less authenticated or less verified than the primary? This is where most of the real damage lives. Chains almost always degrade downward — verified and encrypted first, raw address with nothing second. If the system is willing to accept a weaker connection when the strong one fails, an attacker's job is now just to make the strong one fail. Which is usually trivial.

4. What is the failure mode we are actually buying off? Be specific. My real problem was a machine going to sleep. The honest fix for that is power management and a health check that tells me loudly, not teaching my pipeline to go address-hunting. Most fallback chains are built to work around a root cause the team has decided is too annoying to fix. Name the root cause out loud and see if it still looks too annoying.

5. How would we know if the fallback path is being used every day? An unmonitored fallback becomes the primary path within weeks and nobody notices. If you cannot answer this with a metric or a log line, you are not adding resilience — you are adding a second, unobserved production system.

Question four is the one that saved me. My answer was "the machine sleeps," and once I wrote that down, three network addresses to work around a power setting looked as absurd as it was.

why this bites harder here

Two reasons this pattern is sharper for the teams I work with in the Gulf than the vendor literature admits.

First, lean teams. Most of the businesses I advise here run serious operations on four to eight technical people, and there is no separate security function. The person fixing the outage is the person who would have reviewed the fix. There is no second desk it crosses. That is not negligence — it is arithmetic. But it means the "expand the trust boundary" half of the change gets no independent look, ever, unless the same person deliberately puts on the other hat. Which is exactly what I had to do at eleven at night, arguing with my own eight hours of work.

Second, the self-hosting trade. A lot of what I build in this region is deliberately local rather than cloud, because of data residency rules, client contracts, or a straightforward preference not to send sensitive material to a third party — I laid out that trade-off properly in Cloud vs. Local Models in 2026. It is usually the right call. But it comes with a bill nobody itemises: local infrastructure has no automatic recovery. No managed failover, no someone-else's-on-call, no SLA. The privacy win and the reliability loss are not two decisions. They are the same decision, seen from two sides. "Free" local infrastructure costs you uptime, and the moment you try to buy that uptime back with clever fallback logic, you start spending the privacy you self-hosted for in the first place.

That is the trap. You self-host to shrink the trust boundary, then patch the resulting fragility by widening it again. The same trap in reverse — shipping capability faster than you can review it — is what I cover in Shipping Faster Than You Can Secure.

what I do now

I treat every fallback, retry-to-a-new-endpoint, and redundancy change as a permissions change, and I run the five questions above before a line of it gets written. It is now standard in my engagements: any change to the set of addresses an automated system will contact unattended goes through the same review as granting that system a new credential. Because structurally, that is what it is.

If you have an AI pipeline or an automation stack that has quietly grown fallback paths over the last year — and most of them have — it is worth an afternoon to map what your systems will actually try to connect to when things go wrong. That mapping is exactly what I built after reverting my own fix, and it is a standard part of a consulting day: a map of the real trust boundary, and a list of the places resilience has been paid for in access. Usually two or three surprises in there. If you're running automation for your business across the UAE or anywhere else, book a consulting day or send me an inquiry first if you'd rather talk before booking.