8,600 Green Tests, 31 Real Bugs

An AI agent told me the rebuild was done — built, verified, fully tested, with more than 8,600 backend tests running green. Three independent reviewers then found 31 real problems in it. Two of them would have shipped into production and quietly cost me money for weeks.

The best bug of the lot was sitting inside the safety feature I had just built to prevent the incident that started the whole thing. I'll get to it.

Why I rebuilt a working checkout

One of my businesses takes payment online. When I tore its checkout apart this summer, the checkout worked fine. That was the problem.

The way it was built, a single order could produce three or four separate card charges within a couple of minutes: the initial charge, then every optional extra the customer accepted on the way through, each one its own transaction. From an accounting point of view, harmless. From a card-network point of view, I had turned one customer into three or four separately disputable transactions.

Card networks judge you on the ratio of disputes to transactions, and an annoyed customer does not dispute one line item — they dispute the whole experience, which now means several separate charges pointing at your merchant account. Any card-not-present business that lets an order grow after the first click is doing this to itself. Nothing in the daily numbers tells you: the revenue looks identical either way. The exposure does not.

So the rebuild had one goal: one order, one disputable transaction. Authorize the customer's card once at checkout, hold the funds without taking them, grow that same hold as extras are accepted, and capture the money later in one movement. It's the hotel model — one hand on your deposit at check-in, one settlement at the end.

I dictated the problem out loud and locked the design section by section, signing off on each before anything was written: authorize instead of capture, increment the same hold per extra, fall back to a separate charge only when incrementing isn't possible, route refunds as either a real refund or a pre-capture reduction of the hold, and capture on one of three triggers — someone on my team pressing a button, a completion signal arriving from another system we use, or a safety-net scheduled job. That job runs on a five-day deadline, chosen against a seven-day worst-case authorization window, so nothing expires while I'm asleep. A ledger table is the single source of truth for what was authorized, captured and reduced.

Then I launched it straight into production behind environment variables. No phased rollout, no shadow mode — because I had decided the review process was going to be the safety net.

Good design. Signed off. Implemented. Reported as done.

Three reviewers, three angles of attack

I don't accept "done" from the process that wrote the code. I ran three adversarial reviewers in parallel, each independent of the implementer and of each other, each with a different mandate:

• Money safety. Can a customer be charged twice, charged the wrong amount, or refunded more than they paid?
• Spec compliance. Does this do what was agreed section by section, or something adjacent that looks similar?
• Test adequacy. What could be broken here that would still make every test pass?

Three narrow mandates rather than one instruction to review it carefully, because a reviewer with a general brief produces a single kind of attention — usually whatever the code most obviously invites you to look at. A narrow mandate forces the pass to hunt one class of failure and to keep hunting after the code has stopped looking suspicious.

Between them: 31 real issues. Not style notes. Two were launch blockers, and both are worth describing, because neither had anything to do with the payment logic itself.

The first: every order that was authorized but not yet captured would have reported zero revenue to my analytics. The purchase event fired, correctly, on schedule, with a value of nothing. Every channel would have looked like it had stopped producing money.

The second: extras charged through the new incremental path stopped firing purchase-tracking events entirely. Not wrong values — no events at all.

Both are invisible in production for a long time. Nothing breaks, nothing pages anyone, nobody complains. The platforms I spend money on would have kept spending against a picture in which the revenue had evaporated.

Why 8,600 green tests caught neither

Here is the uncomfortable part.

The tests were not bad. They exercised the new logic correctly. They asserted that the event fired, that it fired once, that it fired at the right moment in the flow.

Nobody had written a test asserting that the dollar amount in that event was not zero.

That's the whole gap. A green suite proves the code does what its tests expect. It does not prove the tests expected the right thing. It is not a window onto reality — it's a mirror showing you your own assumptions, in high resolution, in green. When the same process writes the code and the tests, the assumptions are shared, so the blind spot is shared too. More unit tests from that process would have produced more green.

This gets worse with AI in the loop, not better. A team that generates code and tests together can produce thousands of assertions in an afternoon, and the sheer volume is persuasive: 8,600 green tests reads like diligence. It is diligence about everything the author already thought of, and nothing else.

Notice the shape of both blockers. Neither is a mistake inside the logic that was specified. Both sit at the seam between the rewritten part and a part nobody raised in the design sessions — where an implementer's attention runs out, and where its tests stop too.

The question that finds these bugs isn't "does this work?" It's "what would make this technically pass and still be wrong?" That's a different discipline, and it has to be run by someone — or something — that did not build the thing. It's the same reason I now commission audits outside the frame of the question I actually care about: the seam between two correct pieces is the one entry a scoped checklist never has.

Then I reviewed the fixes

Round one produced fixes. The obvious move is to ship them.

Instead I ran a second, focused audit of nothing but the fixes, on the principle that a change to money-handling code is itself a risk, and a change made in a hurry under the shadow of a blocker is a bigger one. Repairs get written with one failure in mind; the rest of the system is out of frame.

Six more real bugs, introduced by the repairs. One was a scenario in which a customer could end up captured twice — exactly the failure the whole project existed to eliminate, reintroduced by the process of eliminating it.

If you take one operational habit from this post, take that one. Patches to anything touching customer money get their own review pass. Every time.

The best bug was in the safeguard

A day later, an uncaptured authorization turned up on an account with no saved payment method at all. Dangerous state: nothing left to charge if the hold expired or a retry was needed, and my own rule said it should be impossible.

Forensics using the payment processor's own event log — not my application logs, which only knew what my code thought had happened — showed the customer had removed their own card after a run of declines. The single guard on that action turned out never to apply to the cards that mattered most.

The proposed fix was thorough and technically excellent: a multi-condition guard enumerating every state in which a card could be load-bearing — open authorizations, pending retries, and the rest — and blocking removal in each. It took 12 confirmed fixes across money safety, spec and test adequacy before it passed its first adversarial review.

Then the review found a case it still got wrong. Specifically: the case it had been built for.

So I threw it out and proposed what a non-technical founder says in a meeting: a customer can never delete their last payment method. Full stop. One sentence. No enumeration, no state analysis, no list of exceptions to keep current.

It passed the second adversarial review with zero blocking findings and deleted most of the complexity in one line of business logic. The final version shipped with 886 payments-module tests green, after three generations of the same pull request had been opened and closed to get there.

The lesson generalizes past code. When a fix requires you to enumerate every case the system could be in, the enumeration is the risk. A fence with a list of gates is only as good as your memory of the gates. A rule with no exceptions has nothing to forget.

What to take from this

Three things, none of them about payments specifically:

1. A passing test suite is an agreement between the code and its author's beliefs. For anything touching customer money, revenue reporting, or numbers you would have to defend in an audit, that agreement is not evidence. Independent adversarial review is a separate discipline, not more of the same activity.
2. Fixes need re-auditing. Round two found six real bugs in round one's repairs, including a double-capture path.
3. Prefer the one-sentence invariant to the clever case analysis, and notice when your engineers — or your AI — reach for the second. The blunt founder rule beat the sophisticated guard, provably, under review.

Plenty of Dubai and wider UAE businesses are now pointing AI agents at the systems that actually run the place — finance operations, customer support queues, procurement approvals — code that was written once, works, has never been attacked by anyone whose job was to break it, and is now being changed faster than anyone can read it. The same attack-the-whole discipline is what catches cross-cutting defects in a big agent-driven migration, as ten adversarial reviewers on a legacy rewrite showed me.

I rebuilt a checkout that worked, and the only reason it stayed working is that I paid people — well, agents — to break it first. If you have code touching customer money that nobody has independently tried to break, or you're rolling out AI-assisted development and want a review discipline attached to it first, that's a good use of a consulting day. Book a consulting day or send me an inquiry first if you'd rather talk before booking.