The 'Fast' Model That Thinks in Secret

I added a second model to the machine in my office for one job: read an email and answer with a single word. PROMOTIONAL. PERSONAL. Nothing else. A small Gemma model, 26 GB on disk at eight-bit precision, running beside the large model that does the heavy work — on paper, the cheapest and fastest thing on the box.

My first sanity check capped the reply at ten tokens — generous for one word — and sent an email.

It came back empty. Not an error, not a refusal: nothing in the answer field at all, because the model had spent all ten tokens thinking privately about a one-word question. Raise the cap to three hundred and it answers correctly. In that gap it was burning 150 to 200 tokens of invisible deliberation before every classification.

The answer was three tokens. The thinking was two hundred.

With the reasoning off and the thing in production, I measured what the job actually costs: an average of 3.0 tokens per email. Three tokens — the entire useful output of this system.

To produce those three tokens it had been generating 150 to 200 first — reasoning the caller never sees, never stores and cannot use. Roughly fifty to eighty times the latency the task requires: about 300 milliseconds an email afterwards, against something on the order of twenty-five seconds while thinking was on. That second figure is inferred from the ratio I measured, not a stopwatch reading of the old setup. The 300 milliseconds is measured. The tax is real either way.

At 300 milliseconds warm, this model clears roughly 12,000 emails an hour on hardware I own. With the hidden reasoning on, the same hour lands two orders of magnitude below that. Same model, same machine, same accuracy — one default setting between a queue that keeps up and a queue that never will.

"Fast" is a configuration, not a property of a model

I chose this model because it is small and quick. Small and quick is what it is capable of, not what it does when you install it. Its default was to reason internally, and nothing in the model's name or size tells you that.

This is the normal failure with this generation of models. Reasoning modes are a genuine advance, and vendors default them on because on hard questions they look better. Your workload is not a benchmark question. Sorting an email into one of five buckets needs no chain of thought, and you pay for one anyway — in seconds on your own hardware, in billed tokens on someone else's.

The lesson for businesses: when someone quotes you a speed or a price per call, ask what the model was doing while that number was recorded, and whether the default configuration matches it. Treat every number, the flattering ones included, as a claim rather than a fact. The cheapest call is the one that does the least work, and most models will not do the least work unless told to.

An off switch you have not verified is not an off switch

I turned the reasoning off. It stayed on.

The shape of this is worth understanding even if you never touch the stack. The model's template turns thinking on if any of three things is true: an explicit flag is set, tool definitions are attached, or the first message carries a system instruction. That last one fires on every request here — a production classifier always opens with a system instruction. The feature was on by structure, not by anyone's choice.

And the flag I set first was not the flag the model reads. The real parameter name differed from the obvious one by a single word — close enough to look correct in a config file, different enough to do nothing.

So verify at the output, not the setting. I only knew reasoning was off when tokens per email dropped from the low hundreds to 3.0. Until you see the change in a number that matters, you have changed a file, not a system.

A correct fix that looks broken will get thrown away

Then the part that cost the most time, and it had nothing to do with models.

I edited the service configuration, restarted it, tested. No change. Edited again, restarted again, tested again. No change. Two clean attempts, same stale behaviour — which points hard at one conclusion: the fix is wrong, rethink it.

The fix was right. The restart was the problem. The old process had not finished shutting down when the replacement tried to start, so what kept serving traffic was still on the previous settings. Only stopping the service, waiting for it to be gone, then starting it, picked up the edit.

That is an ops race, not an AI problem, and one of the most expensive false leads there is, because it fails quietly. It returns the old behaviour while you conclude your correct diagnosis was wrong and go hunting for a more complicated explanation you do not need.

If your deployment process is "change the setting and restart it", check what happens when the restart overlaps the shutdown. Almost nobody tests that boundary.

Make reasoning a decision per call, not a default nobody chose

The final production battery: 8 for 8 correct, 300 milliseconds average latency, 3.0 tokens per email. Accuracy did not drop when the thinking went away, because the task never needed it.

I did not delete the capability. Reasoning is still available on any request that asks for it, documented for the ambiguous emails where a considered answer is worth the wait. What changed is who decides — the caller, per request, rather than a template default applied to everything.

Scope this honestly: one model, one narrow task, my own hardware. A harder job might well earn the reasoning tokens every time. The principle transfers anyway — expensive capability should be opt-in, not on everywhere because it flatters a benchmark score.

The bottom line

A model sold as fast can be slow and expensive by default, because the cost is spent on tokens you never see. Check what your "simple" calls actually generate before you believe anyone's speed claim.

Then check that the fix you deployed is the one that is running. Here, the difference between those two states was 300 milliseconds and twenty-five seconds, for identical answers.

For the engineers

The model's chat template enables thinking if any of three conditions hold: an explicit flag, tool definitions present in the request, or the first message carrying a system or developer role. The third fires unconditionally for a classifier with a system prompt, which is why reasoning survived the first disable attempt. The second cause was the parameter name — the template reads enable_thinking, the request was sending thinking. Unknown keys are ignored silently, so the config looked correct and did nothing.

The fastest diagnostic is a starvation test. Set the output cap to 10 tokens and send a request whose correct answer is one word. If the response comes back with a populated reasoning field and empty content, the budget is going somewhere you are not reading. At 300 it returned the correct single-word label, confirming the model was right and merely expensive. Steady state after the fix: 3.0 output tokens per email, 300 ms per call warm, 8 for 8 on the accuracy battery.

On the deployment race: editing the daemon's config and issuing a stop-then-start did not reload the value, twice, because the outgoing process had not released ownership when the incoming one launched — so the old settings kept serving. Stop, poll until the process is gone, then start. And assert on the served configuration afterwards, not on the file you edited.

The default-routing problem has a sibling: Your Cheap Path Is Running on the Expensive Model is what it looks like when the fallback, not the template, decides which model answers. And The $370 AI Bill: A Cost Audit Post-Mortem is how I audit this class of silent waste when it arrives as money instead of latency.

If your AI features feel sluggish and nobody can say where the seconds go — or your token bill is growing faster than your usage — book a consulting day for it: I will measure what your calls actually generate, switch off the work you are paying for and do not need, and leave you with a per-request policy for the cases that genuinely deserve deliberation. Book a consulting day or send me an inquiry first and I'll send back the starvation test to run yourselves.