What You'll Still Be Running in Three Years

I needed to add proper search to a system in my Dubai office — the kind where you ask a question in ordinary language and get back the three documents that actually answer it. That requires an embedding model: software that reads each document once and turns it into a long list of numbers that stands for its meaning.

Those numbers are the part nobody warns you about. Every document you index is stamped by whichever model did the reading, and a stamp only compares to other stamps from the same model. Change the model and every number you have stored becomes noise.

So the question I asked was not which model scores highest this month. It was which one I would still be content running in three years. The model I chose scores 70.58 on the standard multilingual benchmark — the top open-weight result at the time — and that score is not why I chose it.

The model swap is an afternoon. The re-indexing is the project.

Replacing an embedding model, as software, is trivial. Download, load, point the service at it, done before lunch. What is not trivial is everything downstream: every document you have ever indexed has to be read again, by the new model, and every stored vector thrown away and rebuilt.

For a small pilot that is an inconvenience. For an organisation with years of contracts, tickets, reports and email, it is a project with a budget, a schedule and a risk register. The software is free to change. The data is not.

The lesson for businesses: the switching cost of an AI component almost never lives in the component. It lives in whatever the component has already touched. Before you ask which option is best, ask what it would cost to be wrong — and where that cost would land.

I chose the option with an exit, not the option with the score

There were two sensible shapes on the table: a smaller, compressed version of the model that would use noticeably less memory, and the larger one at full precision.

I took the larger one, for a reason that appears on no leaderboard. It supports truncation. It emits up to 4,096 numbers per document, and you can cut that list shorter afterwards — to 1,024, to 256, as far down as 32 — at the moment you read it, without going back and re-reading a single document.

That is an escape hatch for the exact failure I was worried about. If storage gets expensive in two years, or search gets slow, I can shrink every stored vector without touching the original text. The compressed model would have saved me memory today and left me with no such option later.

The licence is the second exit: Apache 2.0, so the terms cannot be rewritten underneath me by a vendor who has discovered pricing.

That is the line between a decision and a commitment. Optionality you may never exercise is worth more than a margin you can measure today, whenever the alternative is unwinding your entire corpus.

The expensive-looking option was not the expensive one

The larger model needs roughly 15 GiB for its weights and peaks somewhere between 15 and 20 GiB while it is working. That sounds like the reason to take the compressed version.

It is not, because of what else is on that machine. The same box holds a language model occupying 398 GB. Against that, the difference between the two embedding options is a rounding error — I would have been economising on a resource that was not the scarce one.

This is the most common bad trade I see in AI infrastructure: a real saving, carefully argued, taken out of a budget that had room, in exchange for flexibility that had none. Cheaper is only cheaper when the thing you saved was actually in short supply. Check which of your constraints is binding before you optimise against it.

Test the exit on the first day, not the day you need it

A feature you have never exercised is a claim, not a capability. So before anything went into the index, I checked the two properties I had actually paid for.

The first was language. The model handles 100+ languages, and in Dubai that is not a nice-to-have — a contract in Arabic and its summary in English are the same document, and search has to know that. Matched English–Arabic and English–French pairs came back with similarity scores of 0.79 to 0.83. Two unrelated pieces of English text sat around 0.53. The model is genuinely ranking meaning above language, not quietly clustering by alphabet.

The second was the escape hatch itself. I took the full 4,096-number output and cut it to 1,024. Similarity held at 0.728 instead of collapsing. The exit works, and I know it works now rather than during the emergency that would make me need it.

The generalisable point: when a vendor tells you migration is supported, ask whether anyone has done it, and ask to watch. An untested migration path is marketing.

Sort your decisions by how expensive they are to undo

While I was setting this up I asked myself whether the work should run on the big machine in my office or on my laptop. I gave it about five minutes, because it is completely reversible. If the answer is wrong I move the service on a Tuesday, and nothing already stored cares.

The format of the stored data is the opposite kind of question. Get that wrong and the fix is re-reading everything you own.

Most teams spend their deliberation backwards. Hours of argument about hosting, tooling and which provider to call; ten minutes on the schema, the vector format, the model that stamps the archive. Put your care where the undo button is missing.

Take this as a snapshot and nothing more: a better model will exist within a year, probably sooner, and its benchmark score will beat 70.58. I expect to still be running this one anyway, because the cost of moving was never the download.

The bottom line

Pick infrastructure for the exit, not the entrance. A benchmark score describes a model this month; a data format describes your obligations for years. When one decision is cheap to reverse and the other is not, they do not deserve the same amount of your attention — and it is almost always the boring one that locks you in.

For the engineers

The property that decided it is Matryoshka representation learning: the model is trained so that the leading coordinates of the embedding are themselves a usable embedding. Truncate 4,096 dimensions to 1,024, renormalise, and you keep most of the geometry — measured on my own pairs at 0.728 cosine after truncation, against 0.79–0.83 at full width, with an unrelated-text floor near 0.53. Outputs are unit-norm at 4,096 dimensions, so cosine and dot product agree and nothing has to be normalised at query time.

That turns dimensionality from an architecture decision into a read-time parameter. Storage and index size scale linearly with dimension, so a later 4:1 cut is available without regenerating anything — you re-slice what you already have. A model without this property makes the choice permanent at write time, which is a bad place for a permanent choice.

Two more specifics that mattered. Full precision rather than a quantised build, because the quantisation error lands in the same low-magnitude coordinates that truncation depends on. And a 32K context, so long documents are embedded whole rather than chunked to fit the encoder, which keeps chunking a retrieval decision rather than a hardware one.

If you want the concept behind this explained without the jargon, RAG Explained for Executives covers why document search behaves this way at all — and Build vs. Buy vs. Blend is the frame for deciding which parts of this you should own versus rent.

If you are about to choose the model that will read and index your document archive — or you already have one running and nobody has worked out what re-embedding it would cost — spend a consulting day on it first: I will pressure-test the choice against the migration you would rather never do, check whether the exit you were promised actually functions, and leave you with a format you can live with. Book a consulting day or send me an inquiry before you commit an archive to anything.