I Gave an Agent a Browser, Real Credentials, and Fifteen Hours

Two days of fast changes to a feature that moves money left me with a 110-item manual test plan and zero appetite for executing it. A test plan nobody runs is decoration. So I did the thing most people describe as reckless: I gave an AI agent staging admin access, a real mailbox, a third-party sandbox, a browser, and the night.

It ran for about fifteen hours, from 15:34 to 08:40 the next morning, almost all of it unsupervised. By the time I made coffee it had worked through most of the plan, captured 1,749 evidence screenshots, and left five bug fixes sitting as reviewable pull requests.

It also reported two critical production bugs that did not exist. That part matters more than the successes.

The setup: real credentials and a very short list of rules

The system under test was the paid surface of a product I run — the screens where a counterparty commits to something, the amounts get calculated, and the chain of automated emails that surrounds all of it. Exactly the kind of surface where a bug costs real money in both directions.

I handed the agent four things in browser tabs: staging admin, a real mailbox to receive the automated emails, the sandbox of the third-party system that moves the money, and the admin UI of the external system that fires those emails.

Then the rules, written down before anything started:

• No refunds. Not under any circumstance, unless I later said otherwise in writing.
• Email is read-only. Open it, read it, click links in it. Never send, never delete.
• Close every temporary tab you open. I did not want to wake up to sixty orphaned sessions.

That list is short on purpose. Long permission documents don't get followed; they get skimmed. Three rules a competent operator could recite from memory is the right size, and it leaves an obvious edge for the agent to notice when it is about to cross one. I've argued before that your agent shouldn't be holding your passwords; this run was the other half: a deliberately scoped set of keys, for one night.

Everything else was the test plan itself: 110 numbered cases, grouped into phases, each with a stated expected result. The plan existed already. That is the honest precondition — the agent did not invent what "correct" meant. It executed someone else's definition of correct, which is the only version of QA automation I trust.

Who clicks and who judges

The naive way to run this is to point the most capable model you have at the browser and let it grind for fifteen hours. That bill would have been ugly — the same cost discipline I applied when auditing what AI projects actually cost.

What I did instead: the flagship model piloted the first phase itself, hands on the browser, so it learned where the real friction was — which pages load slowly, which selectors are ambiguous, where the flows loop back. Then it delegated the mechanical execution of the remaining phases to a cheaper model, dispatched one phase at a time through a single shared browser, and kept two jobs for itself: verifying results against server-side data through that external system's API and the application logs, and adjudicating pass or fail.

That division is the whole cost model. Clicking is cheap labor. Deciding whether the result was correct is not. One phase alone burned 297,497 tokens and 147 tool calls; if every one of those had run at flagship prices for work that amounts to typing into a form, the run wouldn't have been worth doing.

The other reason the split works: a model that has driven the flow once writes far better instructions for the model that drives it fifty more times. It stops describing the ideal path and starts describing the real one.

Two critical production bugs that were my own laptop

Around the middle of the night the agent reported a P0: the charge button was dead. Clicking it froze the application entirely. Nothing moved.

Earlier it had reported a different critical failure: a form field on the same flow that refused to accept input. Every keystroke went nowhere. The app looked wedged.

Both were false. Neither was in the product.

The frozen button was a native browser confirmation dialog — the kind the operating system draws, not the page. It sat there invisible to the automation tooling, which had no way to see or dismiss it, so the tool call simply hung. The wedged field was a password-manager extension quietly intercepting every keystroke before it reached the input, a bouncer standing between the keyboard and the form.

The agent did not retry around either of them, and it did not shrug and mark them blocked. It root-caused both, live, mid-run — asking me for a single manual click at one point to confirm the dialog hypothesis. Then it fixed the frozen-button case at the source, shipping a change that removed the redundant native dialog altogether.

A meaningful share of what an autonomous QA run reports will be artifacts of the machine it runs on, and those need exactly the same rigor as real defects. An agent that treats environment noise as "just retry" is worse than useless, because it teaches you to discount its findings. An agent that diagnoses the noise is doing the same work a good tester does.

The bug that was real

Among the genuine findings, one was serious: a link in one of the automated emails could be replayed. Someone who had already gone through the flow could open that same link later and be offered — and charged for — something they already had.

That is a duplicate-charge hole in a flow that moves money. It is also the class of bug a human regression pass finds late or not at all, because finding it requires something slightly unnatural: go back to an old email after the state has changed and click it anyway. An agent working a numbered plan at four in the morning has no instinct telling it that is a silly thing to do.

By the end: 84 passed, 2 failed, 24 blocked by environment constraints I couldn't clear at that hour. Roughly a 95% pass rate among the tests that were actually executable.

The boundary moved, and it stopped anyway

Partway through the night I loosened the rules to unblock coverage — refunds became permitted, but strictly through the admin UI, and I approved a database backfill test.

The agent got to that backfill test, looked at what it actually required, and halted. The action meant a real write outside the browser, and it judged that my permission — worded in the context of browser testing — didn't clearly cover it. It stopped and asked rather than inferring consent.

One instance is not a guarantee. But it is the behavior that makes an overnight run survivable: it treated an ambiguously scoped permission as a reason to stop rather than a loophole to walk through. If you grant standing authority to an autonomous system, the thing to test for is not obedience — it's whether it notices when your instruction has run out.

It shipped the fixes, not a report

The part I now consider the point is the part I expected least. Rather than leaving a findings document for the morning, the agent triaged its own confirmed bugs, picked the highest-value ones, and built each fix in its own isolated worktree with tests.

Five pull requests, merged overnight. Six more the next morning out of follow-up questions. Eleven in total. Each one narrowly scoped so it could be reviewed and merged independently, and each with its test suite green before merge — 39 of 39, 95 of 95, 108 of 108, 176 of 176. When one unrelated pull request showed a single failing test out of 5,636 in the full CI suite, it traced the failure back rather than assuming its own change had caused it. The assertion was stale, left over from an earlier merge that had never updated it.

Two of the five overnight fixes were things nobody on the other end would ever report. One was a missing analytics event on a retry screen, silently undercounting the attempts that eventually went through. The other was a step reporting a higher amount to the analytics pipeline than was actually charged.

Nobody complains about that. It just quietly poisons the numbers that spending decisions get made on. "Nothing looks broken" and "the numbers are right" are different claims, and only one of them is checkable.

What made it defensible

The run is worth something because it left a paper trail I can hand to anyone: a full transcript, a verdict per test case with the reasoning behind it, 1,749 screenshots, and a self-contained evidence report. If a finding is disputed, I can replay how it was reached. If a pass is doubted, there is a picture.

That is the real deliverable. One night replaced what would have been a multi-day human regression pass — but it only counts because the output is auditable rather than a confident summary I'd have to take on faith.

The lesson for a business reader is narrower than "AI can do QA." Autonomous execution scales the clicking, not the judgment. You still need a written definition of correct, a short and explicit list of what the system may touch, a human awake enough to widen that list deliberately, and the discipline to root-cause the false alarms instead of filtering them out. Skip any of those and you get a beautiful report full of findings nobody trusts. The close cousin of this run is the question of which specific action an agent should never take at all, which I tackle in Never Click Pay.

I handed an agent a browser and a night, and got a regression pass back before breakfast. Plenty of companies in Dubai and across the UAE ship fast on small teams, where the regression pass is the first thing to get skipped and a bug that touches money reaches a user before it reaches a tester — if that's you and you want to know whether your own release process could hand a night shift to an agent, what to write down first, what never to grant, and what the evidence trail has to look like, that's a consulting day's work. Book a consulting day or send me an inquiry first if you'd rather talk before booking.