Your Cheap Path Is Running on the Expensive Model

I spent a morning working out why the largest model in my office was busier than the work I was actually giving it. The answer, when I finally caught it in the traffic, was a request whose entire job was to write a short title for a chat window.

It was being answered by a model with a trillion parameters. That model lives on a 512 GB M3 Ultra Mac Studio, and it is the most expensive thing on the machine by every measure that counts — memory held, heat produced, and the queue that everything else waits in behind it.

The tool sending that request has a second, separate slot for exactly this kind of work: small, frequent, low-stakes calls that keep an interface feeling alive. A one-line default in a wrapper script I maintain had that slot quietly falling back to the same model as everything else. An earlier sample of 66 requests from the same tool had a median size of 8.7 KB — and my best estimate was that around half of that traffic was this sort of background housekeeping, taking the expensive route by default.

The cheap lane already existed. Nothing was driving in it.

This is the part worth sitting with. Nobody had failed to design a cost-efficient path. The path was there, documented, and supported. A small classification model was already loaded on the same machine, already paid for in memory, already idle.

The failure was one line of configuration deciding what "fast model" meant when nobody had told it. Left unset, it fell back to the main model — the largest and most contended thing available. The setting that was supposed to protect the expensive resource pointed straight at it.

The lesson for businesses: your defaults are your architecture. The routing rules that nobody consciously chose are the ones running the majority of your traffic, and they were chosen by whoever wrote the fallback, on a day they were thinking about something else.

Nothing broke, which is exactly why it lasted

A trillion-parameter model writes an excellent four-word session title. That is the whole problem. There was no error, no failed request, no alert, no red square on a dashboard. Every one of those calls returned a correct answer.

The only symptom was that the big model felt slower than the work I could account for — the vaguest complaint in all of infrastructure, and the one most likely to be blamed on the hardware, the network, or the weather.

This is the shape of the most durable waste I find in client systems. Outages get fixed within the hour because they are loud. Misconfigurations that only cost money are silent by construction: they succeed, every time, at a price nobody is reading. If your only detection mechanism is "someone will complain", you will never find this class of problem. Nobody complains about a correct answer.

Read your traffic, not your configuration file

I want to be honest about the strength of my own evidence here, because it is the same standard I would hold a vendor to.

What I measured directly was small: a single title-generation request, a few kilobytes in total, being served by the production model. What I inferred was larger — that sample of 66 requests with a median size of 8.7 KB, from which I estimated roughly half the volume was background work misrouted the same way. That is an estimate, not a meter reading, and I would not put it in a business case without saying so.

But the direction was not in doubt, and the fix cost nothing: repoint the background slot at the small model that was already running. No new hardware, no new licence, no new service to keep alive. The most profitable ten minutes of that week were spent changing where a default pointed.

Verify at the road, not on the map

A configuration change is not evidence. The only thing that counts is what arrives at the expensive resource afterwards.

So I sampled it: 72 requests over 30 minutes hitting the production model, classified by what they actually were. Zero background tasks in the set — no title generation, no suggestion prompts, no session recaps. All of them were now landing on the small model, which is what it was there for.

That is the check I would ask any team to run. Not "did we set the flag", but "go and look at thirty minutes of real traffic and tell me what is in it". The gap between those two questions is where most of the cost-optimisation work I get called in for actually lives.

Go and audit your own cheap path this week

Three questions, and you can get answers to all three before lunch.

• Which of your AI features have a designated cheap or fast route, and what model does that route resolve to right now — not in the documentation, in the running system?
• What happens when that setting is left blank? If the fallback is your main model, you have this bug until proven otherwise.
• Can you see, per feature, what is hitting your most expensive model in a given half hour? If the answer is one aggregate number, nobody can find this.

The honest limits: my version of this cost capacity rather than cash, because the model was running on hardware I already own. On a metered API the same misrouting arrives as a bill, which is easier to notice and much harder to break down — you see the total, not which of your features chose the premium lane.

The bottom line

Having a cheap path is not the same as using it. The most expensive default in most AI systems is not a model choice anybody argued about; it is a fallback that nobody set, doing exactly what it was written to do.

Go and check where your background tasks actually go. It takes an afternoon, it produces no error messages either way, and roughly half the traffic I look at is in the wrong lane.

For the engineers

The coding assistant exposes a separate model slot for auxiliary calls — title generation, suggestion prompts, session recaps — distinct from the main conversational model. A locally maintained wrapper script set that fast-model variable with a fallback to the same variable used for the primary model. Unset in practice, so every auxiliary call dispatched to the largest resident model on the box.

The cost is not tokens, it is occupancy. Those calls are tiny — a few KB of prompt, a line of output — but each one takes a slot on a memory-bound model whose weights and KV cache already sit near the machine's ceiling. They queue against real conversational work and inflate its time-to-first-token, and they are frequent by design: one per new session, more per interaction.

The fix was a one-line change at the wrapper level, repointing the fast slot to a small classification model already resident on the same host. No additional memory, no new process.

Verification mattered more than the change. Config diffs prove intent, not behaviour, so I sampled the gateway instead: 72 requests over 30 minutes to the production model, classified by request shape and system prompt. Zero auxiliary requests present. Before the change, a directly observed title-generation request had been served by the trillion-parameter model, and a 66-request sample of the same client traffic carried a median size of 8.7 KB — small, frequent, and pointed at the wrong place.

The small model in question is the one from The 'Fast' Model That Thinks in Secret — cheap only after I found and switched off the hidden reasoning tax it shipped with. And for the audited version of this exercise, The $370 AI Bill: A Cost Audit Post-Mortem is the full method.

If you are paying for a large model — in tokens or in capacity — and you have never checked which of your features are actually reaching it, a consulting day pays for itself there: I will sample your real traffic, show you feature by feature what is taking the expensive path, and put the routine work back on something cheap that you already have. Book a consulting day or send me an inquiry and I'll send back the three questions to put to your own team.