Make Your Index Disposable

what happened — three databases, "disk image is malformed"

I opened the home page of a system I built and got a server error. Not a slow page, not a warning. A dead application, on an email archive holding roughly 580,000 messages across a couple of dozen mailboxes. Three of the account databases were reporting the same thing: disk image is malformed. Corrupted. The home page iterates every account, so three bad files took down all of it.

The cause was boring and entirely my fault. I ran a cleanup script from the host machine that wrote directly to those databases while a containerized service was also writing to them. File locking is not coherent across that boundary — the container and the host each thought they had the lock, and both were right, and the files got shredded in the middle.

Here is the part that matters. I lost nothing. Not one message. Message rows, search index, per-account state, all of it came back, because none of it was ever the real data. Four days later I did it again — a burst of rapid container rebuilds killed a writer process mid-write, same corruption, same three-word error. Same outcome. Zero data loss, twice.

That was not luck or good backups. It was one decision made months earlier, and re-argued five times before it stuck.

the fix — the archive is files, the database is a cache

When I designed the thing, the only real question was: what is the source of truth?

The obvious answer, the one every instinct pushes you toward, is the database. Put the messages in tables, index them properly, query them fast. Sophisticated. Queryable. Correct-looking.

I chose plain individual message files on disk instead. One file per message, standard format, immutable once written, readable by any tool on earth including a text editor. The database became something much smaller: a per-account index — a lightweight lookup layer holding what you need to search and browse quickly. Embeddings for semantic search sat one layer further out, derived from the files, and I deliberately deferred building them at all.

The owner of the system pushed back on this at least five times. Wouldn't a proper server-grade database be better? Isn't a real engine the mature choice? Every one of those pushbacks was fair, and I did not defend my original pick as a matter of pride. I laid out the tradeoff again each time, and each time the resolution was the same: separate what is expensive to change from what is cheap to swap. The canonical archive is expensive to change — get it wrong and you are migrating 580,000 files. The index is cheap. If a better engine wins the argument next year, I delete the index and rebuild it overnight from the files.

Which is exactly what recovery looked like. I did not restore anything. I deleted three corrupted files and re-derived them. The scary event became a chore.

the framework — canonical or derived, for every single store

Most teams have this backwards. They treat the database as the crown jewels and everything around it as scratch space. Invert it. Here is what I now run on every system I touch, and it takes about an afternoon.

1. List every place data lives, then ask one question of each: if I deleted this right now, could I rebuild it? Not "would it be annoying." Could I, mechanically, regenerate it from something else I still hold? Databases, search indexes, vector stores, caches, generated reports, thumbnails, summaries. Write yes or no next to each. That single column is your architecture.

2. Anything you can rebuild is not precious — and treating it as precious makes you slow and scared. Teams that guard derived state stop doing the things that keep systems healthy. They avoid schema changes, they avoid reindexing, they avoid replacing the search engine because "we'd have to migrate the data." No, you wouldn't. You'd throw it away and regenerate it. A store you are afraid to delete is a store that will eventually dictate your roadmap.

3. Anything you cannot rebuild deserves paranoid protection and the dumbest possible format. Plain files. Standard, boring, self-describing formats. Not a proprietary engine, not a schema that only your application understands, not something that requires a running service to read. My canonical archive is readable in fifty years by anything that can open a text file. That is the whole point of canonical: it must outlive every tool you currently use.

4. Backup, access, and recovery all change per class — write down which. Canonical gets versioned, off-site, checksummed backups and append-only or write-once access. Derived gets no backups at all — it gets a rebuild script, tested on a schedule, with a known wall-clock time. If you cannot tell me how many minutes it takes to rebuild your search index from scratch, you do not have a derived store. You have an undocumented canonical one, and you are one bad write away from finding out.

5. Never write to a live system's data store from outside that system. This is the one that bit me twice. Not from the host while the container runs, not from a maintenance script "just this once," not from a notebook because it's faster. Everything goes through the running application, or the application stops first. Locking guarantees do not survive a boundary. And while we're here: a background maintenance query of mine, doing an enormous number of comparisons, held the write lock and wedged the entire application twice in one night. If a query is long, it belongs against a copy, not the live store.

I'll give you the counter-example that proves the rule, because it's the one that actually cost me. A container restart re-triggered a migration that was supposed to run once, and it silently rolled back about 2,700 rows of careful manual data repair — hand corrections I had made and never stored anywhere else. Everything derived rebuilt itself perfectly. The only thing I lost was the work that had no canonical home. Human corrections are canonical data. Treat them like it.

why this bites harder here

In the Gulf I mostly work with lean teams — organizations doing serious revenue with an IT function of two or three people, sometimes one. There is no database administrator. There is no on-call rotation. When something corrupts at 11pm, the person fixing it is the same person who was going to present to the board in the morning.

That environment does not reward sophistication. It rewards systems where the worst realistic failure is a forty-minute rebuild instead of a week of forensic recovery and an uncomfortable conversation about what was lost. Sophistication is a headcount bet, and most lean teams cannot cover it.

This matters far more in AI systems than in traditional ones, and I want to be specific about why. AI systems generate enormous amounts of derived state — vector indexes, embeddings, chunk stores, extracted entities, cached model outputs, generated summaries. It feels valuable, because it cost real money and real hours of compute to produce. It is still derived. If you can regenerate it from the source documents, it is a cache with an expensive invoice attached, and the moment you start protecting it like a system of record, you will refuse to change your chunking strategy, refuse to upgrade your embedding model, and end up frozen on a 2025 architecture in 2027. I have watched exactly that freeze happen to teams who did nothing wrong except mislabel one folder. The way caches quietly betray their own framing is a running theme — see The Cache That Vanished Every Restart and The Migration Framed as a Feature.

Ask the question. Cheap to ask, expensive to skip.

where I've landed

I now run a canonical-versus-derived audit as a standard first step in every engagement, before I touch a line of anything. It is usually a one-page table, and it is usually the most uncomfortable page in the project, because it tends to reveal that the thing everyone treats as the system of record is actually a derived index — and that the real irreplaceable data is sitting in a spreadsheet on somebody's laptop.

If you're building anything with AI on top of your own data, that table is worth an hour of your time whether or not you ever speak to me. I designed my own archive this way and then had the design tested by two corruption events that cost me nothing — that's the kind of stress-testing a consulting day puts your architecture through. If you'd rather have someone go through it with you and tell you honestly which of your stores you're allowed to stop worrying about, I'm happy to do that in a single call. Bring the list of places your data lives. That's all the preparation it needs. Book a consulting day or send me an inquiry first if you'd rather talk before booking.