Five Research Agents Found My Own Documentation Was Wrong

Before I write code for a step I don't control end to end, I spend an afternoon trying to prove myself wrong. I did that recently for a hand-off step inside one of my businesses, and the most valuable thing the research turned up had nothing to do with the vendors, the rules, or the competition.

It was that my own engineering notes were confidently, flatly incorrect.

The most expensive bug in the project was already sitting in my repository, written down as a fact.

The problem shape

One step I depend on runs inside a third-party system I don't control, and it completes out of band. Not by email, not through an external web portal, not over an API — through a slow offline channel that ends at whatever intake point I nominate on the submission. A product I run, meanwhile, promises the customer a clean digital experience: the thing they are waiting for shows up in their account and they download it.

Somewhere between those two facts sits a gap, and nobody on my side could give me a verified account of how that gap was actually being bridged. Not a bad account. No account at all.

That is the exact moment most teams start building. You pick a vendor that looks right, write the integration, and find out what you didn't know six weeks later when a customer asks where their result is. I've paid that bill. So I spent an afternoon on research instead — but not the way research usually gets done.

Five agents, run at the same time, split by domain

The normal shape of "let's research this first" is one person opening twenty tabs for three days. That's why executives hear the phrase as a delay tactic. It usually is one.

I ran it as a fan-out instead. Five background agents, launched simultaneously, each owning one domain:

• The rules. What governs a third party acting on your behalf inside that channel, and what it is and isn't permitted to do once something is in its hands.
• The primary source. The counterparty's own published statements about how and when the result comes back, read off their pages rather than from anyone's summary of them.
• Competitors. How others in the same category actually close this gap, as opposed to how they describe it.
• Vendors. The landscape of services that handle intake and digitisation on your behalf, and what each one really does once you get past the homepage.
• My own codebase. A full scan of the infrastructure already sitting behind this path: what was built, what was configured, and what was merely assumed.

That last one is the one most people skip, and it is the one that paid. I treated the internal repository scan as an equal-weight research agent, not as background context. External research asks what is true in the world. Internal research asks what my system believes is true. Those two diverge more often than anyone is comfortable admitting, and the divergence is invisible from inside the system, because the system keeps behaving exactly as its wrong assumption tells it to.

Running the five in parallel isn't a productivity trick. It's a design decision about independence. Each agent reached its conclusions without seeing the others' work, so when two of them collided, the collision was signal rather than one thread contaminating another. Hand one agent all five briefs in sequence and it will quietly harmonise them: the first thing it reads becomes the frame, and the contradiction you most needed to see gets smoothed into a subordinate clause. Isolation preserves disagreement, and disagreement is the product of a diligence pass.

Every claim carries a tag or it doesn't ship

Most AI research output is useless in a real decision because it reads as one uniform beige paragraph. A verified primary source, a vendor's marketing claim and the model's own guess all arrive in the same confident tone.

So I made tagging non-optional. Every factual claim had to carry one of three labels:

1. FACT — verified directly against a primary source, with the link.
2. REPORTED — a secondary source says so, with the link, and nobody confirmed it independently.
3. ASSUMPTION — nobody verified this; here is the reasoning behind it.

Three labels. That's the whole framework, and it changes what the document is for. A tagged report can be handed to an adviser or an ops lead who goes straight to the ASSUMPTION and REPORTED lines, because those are the only places the thing can be wrong in a way that hurts. Everything marked FACT has a link they can click.

It also changes how you read your own project. The ASSUMPTION lines are a to-do list of the cheapest verification work available, ranked by how much of the plan collapses if the assumption is false. Without the labels, that list doesn't exist and the risk sits spread invisibly across every sentence.

Untagged AI research is a chat summary. Tagged AI research is due diligence.

The two findings that would have become shipped bugs

The first collision came within minutes. One of my internal engineering files stated that the completed result comes back by email. The agent reading the counterparty's live pages found the opposite: email is not a return channel at all, and the offline route is the only one. My own documentation was wrong, and every downstream decision that had quietly leaned on it was wrong too.

If I had built from my notes, I would have written an inbox parser for an email that never arrives.

The second was worse, in the boring way that operational problems are worse. The intake point on file — the one that decides where the result ends up — was a placeholder. Not a wrong destination. A documented stand-in, with nothing behind it. Something gets handed off, arrives somewhere, and nobody opens it.

Neither is a hard problem once you know about it. Both are catastrophic if you find out from a customer.

Round two: you're paying for the staffing, not the map pin

The first pass recommended a specific vendor. Before wiring anything up, I sent two more agents at it — one to crawl the entire developer documentation, one to sweep product, pricing, terms and company background.

The documentation crawl went through 42 pages in full, including markdown-only variants of pages that don't show up in normal browsing, and inventoried 27 API endpoints. Nobody reads a vendor's docs that way by hand. That's the point.

Buried in there was the fact that killed the original recommendation: most of the vendor's intake locations are not where the work happens. What arrives there is consolidated and moved to processing centres elsewhere. Choosing the location that looks obviously correct on a map added an estimated two to three extra business days versus one co-located with a processing centre. Same vendor. Same price. Slower.

That fact appears nowhere in the marketing pages. It was visible only because something read every page instead of the sales pitch.

When you can't answer the pushback, re-derive

My reaction to that finding was not gracious. If the location isn't where the work happens, what exactly are we paying for?

The useful part is what happened next. Rather than defending the original pick, the reasoning got walked back to the foundation, and the answer came out clean: what I was buying was staffed intake and digitisation as a subscription. Not proximity. Nothing on my side cares where that intake point physically sits, so geography was never the product at all. Once that was clear, the correct configuration of the same vendor changed — a different location, co-located with a processing centre, faster for identical money.

When a sharp "what are we even paying for" can't be answered in one sentence, that's not a communication problem. It's a sign the recommendation was assembled rather than derived. Push on your own decisions the way a hostile customer would, before a hostile customer does.

What came out the other end

The output wasn't a chat transcript. It was a single self-contained report, about 68 KB, with citations throughout, a decision matrix comparing the options, and a full operational procedure for the manual steps — assembled in under 30 minutes of synthesis once the agents finished.

The format matters as much as the parallelism. A tagged report with a decision matrix can be audited by someone else without re-running any of it; a chat thread can't. It is also why synthesis stays supervised: merging five independent findings is where a model is most tempted to smooth a contradiction into agreement, and the contradictions were the point.

Total elapsed: an afternoon. Compare that to the version where you skip the diligence because it "takes days", build on a stale internal note, and discover the placeholder intake point in production.

The bottom line

Two lessons, both cheap to adopt.

Your internal documentation is a source that needs verifying, not a foundation you build on. Stale notes don't announce themselves. They read exactly like current notes, in your own team's voice, which is what makes them dangerous. Point research at your own repository and wiki with the same suspicion you'd point at a vendor's homepage — the same read-only codebase scan that anchors how I reconcile a rewrite against the real system it replaced, and the same "map before you build" instinct behind sending two agents to read the code before a single design question.

And parallel, source-tagged research changes the economics of due diligence. It stops being a multi-day detour that gets cut under deadline pressure and becomes an afternoon that happens before every meaningful build decision. The cost of checking is now low enough that not checking is a choice you are making on purpose.

That matters here specifically. Companies in Dubai and across the UAE run on long chains of outside parties — vendors, outsourced operations, platforms, support desks staffed in another time zone — and in every one of those chains the marketed version and the operational reality differ. A two-day routing penalty or a hand-off point nobody staffs only surfaces when a customer goes looking.

I turned a "we should really research this first" detour into an afternoon, and it caught two bugs that would have shipped. If you have a build decision queued behind exactly that sentence and want the pre-build pass done properly — five threads, tagged sources, a report your own adviser can check — that's a good use of a single consulting day. Book a consulting day or send me an inquiry first if you'd rather talk before booking.