The Migration Framed as a Feature

I pasted a screenshot into my coding assistant and asked the local model what was wrong with the chart. It refused — not "I can't quite make that out", but a flat rejection written by the software rather than the model: only text is supported.

The model has eyes. They ship in the file. The engine I ran it on had been throwing them away: 333 of the model's weights — the ones that handle images — dropped at load, every time the service started, on purpose.

There was no setting to switch back on. The only route to image support was a different engine entirely — so the small feature request was a platform migration filed under a smaller name, and the thing genuinely at risk had nothing to do with images.

"Just add image support" was a platform migration

The engine I had been running is text-only by design. Its loader inspects every weight in the model file and discards anything belonging to the vision half before the model can answer a single question. That is a decision, not an oversight — which is why no configuration on my side could undo it.

The alternative was not that project with vision added. It was a separate open-source project: different authors, different internals, doing the same job a different way underneath.

That distinction is the whole post. Most of my engineering time on that machine had gone into one unglamorous thing — making the model stop re-reading conversations it had already read. A 27,800-token conversation takes 14.0 seconds from cold before the model says its first word. All of that work lived in the engine I was about to replace, and none of it moved with me.

The lesson for businesses: when the answer to a small feature request is "we'll have to change the engine underneath", the feature stops being the risk. Everything else that was quietly running on that engine becomes the risk.

Measure the move against your own baseline, not the new thing's demo

So I did not migrate and then find out. I ran the same three tests on both engines, on the same conversations, before committing.

Asking again inside a live session: 0.5 seconds on the new engine against 0.3 on the old. Asking again after a full restart: 0.6 against 0.5. That 27,800-token conversation, restored from disk: 0.6 seconds against 14.0 from cold.

A shade slower in two places, identical where it counted. The new engine also understood, with no code written on my side, the exact shape of request my coding assistant sends — streaming replies, several tool calls at once, the model's own reasoning blocks. That mattered more than the tenths of a second, because adapter code is the part of a migration that never ends.

The lesson: a migration decision is made on your own tasks, timed the way you already time them. A vendor's benchmark tells you what that vendor is good at.

The number that nearly killed it was a default nobody chose

Then I put it into real daily use, and reuse collapsed.

I knew because I had a figure from before: across 123 real lookups on the old engine, 93.5% reused earlier work, with only 2 outright misses. After the switch, every conversation behaved as though it had never met me — twenty seconds of the machine re-reading everything before a single word came back. It looked exactly like the fear I started with: the caching investment did not transfer.

I asked my AI assistant, working alongside me, what had changed. It answered at once, and the answer was architectural: the new runtime simply doesn't do what the old one did. Structural. Nothing to fix. That was wrong — and so were the next two explanations it offered, each confident, each killed by a single measurement.

The second theory: a hard ceiling on how much of a conversation can be held, so longer sessions fall off the end. The test wrote itself — I sent a 60,000-token prompt and watched. It cached without complaint. The third: a safety mechanism that refuses to restore a saved conversation when memory is tight. Plausible — I have had memory trouble on this box before. The test was to count how often it had fired. Zero. Not rarely. Never.

What was actually happening: the new engine evicts cached conversations by count rather than by size, and ships configured to keep two entries. And "entry" did not mean one per conversation — it meant one per conversational turn, so any two background calls between my messages (a sub-task the assistant runs on its own, a small call to name the session) evicted the entry that mattered. Every message then paid full price to re-read the whole history. One default, raised, and the behaviour came back.

My own tests could not have found this

Here is the uncomfortable part. I had tested. My tests passed.

They passed because they fired requests back to back: the same conversation twice in a row, nothing in between. Nothing in between is precisely the condition under which a two-entry cache never overflows. The test was structurally incapable of producing the failure, however often I ran it. A green result meant only that I had asked an easy question.

What found it was replaying my own captured requests — the real sequence, background calls sitting where they actually occurred. It reproduced on the first pass. After the fix, those same replayed requests came back in 1.0 second and 0.6 seconds instead of roughly 20.

The lesson: if your testing sends work in a tidy sequence, it will never surface failures caused by untidiness — and untidiness is what production is. Record real traffic and replay that instead of synthetic back-to-back calls.

And a second lesson in the same incident: when an AI assistant explains a regression, it produces the most plausible story consistent with what it can see — and what it can see is the system now, not how it behaved when it worked. Plausible is not true. The record of my own past performance — 93.5% for weeks — was the only participant in the room that could not be argued out of its position. Treat every explanation you are handed, by a person or a model, as a hypothesis with a test attached.

I caught all of this only because I had a number from before the change. Without a baseline of your own, a regression after a migration reads as "the new one is just like that" — and you live with it, at a cost nobody writes down.

The feature you moved for has a running cost the demo never shows

While measuring, I found something I had not gone looking for. Once an image appears anywhere in a conversation's history, every later turn re-reads the whole thing from scratch — about 8.9 seconds per turn, against 0.4 to 0.5 seconds for a conversation that is text only. Twenty times the wait, for the rest of that session.

It is architectural, and true on both engines, so it was never an argument against moving. It is an argument about how the feature gets used: one screenshot early in a long session taxes every question after it. The fix is behavioural — put images in a fresh conversation, keep the long one clean.

The lesson: new features carry running costs, and demonstrations only ever show you the first turn.

Don't keep a system alive because you helped build it

The last decision was the one I nearly got wrong.

Earlier that same day I had sent a fix upstream to the maintainers of the old, text-only engine — my work, in public, now part of somebody else's project. So a compromise was available: move the daily-driver model across, keep the smaller, rarely used one on the old engine. Both worlds, and my contribution stays in service.

Priced out, that compromise cost 70 GB of memory sitting idle and 19% of the throughput of the model I use every day, in exchange for sentiment. I retired the old service a few hours later, keeping only its configuration on disk, on the same endpoint, so reversing the migration is one command rather than a project.

The lesson for businesses: the question about a system you already own is never what it cost to build. It is what it costs to keep — and that answer turns unflattering most often when you were the one who built it.

The bottom line

"Just add image support" cost a change of engine, a day of measurement and a retired service. The feature itself worked on the first attempt. Everything expensive sat underneath it.

Migrations rarely fail because the new thing is worse. They fail because nobody wrote down what the old thing was doing well, so nothing raises a hand when it stops.

For the engineers

The text-only server's loader strips every tensor carrying the vision-tower prefix — 333 of them — before the model is instantiated, and rejects any image content block with "Only 'text' content type is supported." No flag exists, because it is not a missing feature. The vision-capable runtime is a separate project with its own prompt-cache implementation, not the first one plus a vision path.

Like-for-like, same conversations, new versus old: warm repeat inside one process 0.5 s vs 0.3 s; across a full daemon restart 0.6 s vs 0.5 s; a 27,800-token history restored from the disk cache in 0.6 s against 14.0 s of cold prefill. It parsed the assistant's wire format natively — streaming, parallel tool calls, native reasoning blocks — with zero adapter code.

The near-miss: cross-session reuse collapsed against a measured baseline of 93.5% over 123 lookups, 2 hard misses. Cause was the new runtime's in-RAM prompt cache evicting by entry count, no byte cap, defaulting to 2 slots — one slot per conversational turn, not per conversation, so interleaved sub-agent calls and title generation evicted the live lineage between user messages. Raising the slot count restored the hit rate. Three hypotheses were falsified individually along the way: a context-window ceiling (a 60,000-token prompt cached cleanly), a free-memory guard (its counter read zero), and architectural incapability (refuted by the historical log).

Synthetic back-to-back requests never interleave and so never evict — that is why my tests passed while production failed. Replaying captured production requests reproduced it immediately; at 32 entries the same replays returned in 1.0 s and 0.6 s. Settled at 32/8, later 64/16. At 128 the disk index's linear scan over entry headers produced a 42 s outlier — the ceiling is a real tradeoff, not free headroom.

Separate and architectural: any image in the history forces a full re-prefill every subsequent turn — ~8.9 s per turn against 0.4–0.5 s text-only — on both runtimes. Rollback stayed a one-command config swap on the same endpoint.

The cache-reuse baseline this migration was defending was built across several posts — Two Hundred Seconds, One Shuffled List and The Cache That Vanished Every Restart cover how the cache works and why it needed a durable tier. And Shipped Is Not Running is the general warning: the migration that demos well on day one is not the migration that survives your Tuesday.

If someone on your team has described the next change as "we just need to add X", and the plan underneath it quietly replaces a database, a model host or a platform your working systems already stand on, that is a consulting day worth having before anything moves: I'll help you write down what your current setup does well before anything moves, put numbers on it so a regression can't hide, and make sure the way back is one step, not a rescue. Book a consulting day or send me an inquiry while the migration is still a proposal.