I Rejected the Better Model With Arithmetic

A new open model landed with the kind of reception that makes you clear your afternoon. Bigger, newer, topping the public comparisons, free to run on your own hardware. I read the write-ups and checked how much disk I had free.

The model I already run does email triage and a pile of other daily work on one machine in my Dubai office — an M3 Ultra Mac Studio with 512 GB of unified memory. It answers at about 21 tokens per second warm on a short prompt. Not fast by hosted-API standards. Fast enough that nobody waits on it.

The question was not "is the new one smarter". It was "will it be faster, or at least not slower, on the machine I actually own". That has an answer you can work out on the back of an envelope, and I worked it out instead of starting a 418 GB download: roughly 15 tokens per second, against the 21 I already had. I closed the tab.

Fitting on the machine and running well on it are two different questions

The candidate's headline numbers read like a straight upgrade: 744 billion parameters in total, 40 billion of them active for any given word, and a context window of a million tokens. On paper it dwarfed what I was running.

The first constraint is whether it fits in memory at all. Models get compressed to fit — the same weights stored at lower precision, which is how anyone runs something this size on a desk machine. The smallest compression practically available put the candidate at 418 to 430 GB. My machine has 512 GB, and that memory also holds the operating system, the working state of every request in flight, and the cache that keeps repeated prompts cheap.

So it fit. Tightly. And "it fits" is where most evaluations stop, which is the mistake. Fitting tells you the thing will load, and nothing about how fast it will answer.

Speed is set by how much a model reads, not how big it is

The whole argument in one sentence: generating each word means pulling a fixed quantity of data out of memory, and your machine can only move so much data per second.

These are mixture-of-experts models — only a slice participates in producing any single word, which is why a 744-billion-parameter model runs on a desk at all. The number that governs speed is not the total. It is that active slice, multiplied by how many bits each weight is stored at.

My incumbent runs 32 billion active parameters at about 3.3 bits each — roughly 13.2 GB read from memory per token produced, at about 21 tokens per second measured. The two numbers are consistent, which is how you know bandwidth is the binding constraint.

The candidate, at the smallest compression that fit, would run 40 billion active parameters at about 3.6 bits each — roughly 18 GB per token. That is 1.36 times more memory traffic for every word. Same machine, same fixed bandwidth. Divide through and you land near 15 tokens per second.

The lesson for businesses: a model that is 36% more expensive to run per word has to be more than 36% better at your task before switching is rational. Not better in general. Nobody in the launch coverage was in a position to tell me that.

Compressing it further would have cost me the stack I had already tuned

The obvious counter-move is to squeeze harder. If the model is too heavy at 3.6 bits, take it to 2. That option genuinely existed: builds at that precision came in between 217 and 239 GB, leaving real headroom on a 512 GB box.

Two problems, both fatal. Published accuracy for those builds sat at 76 to 82% of the model's reported scores — trading away the very quality that made it interesting, to buy back speed I already had. And they existed only in a different file format, which my serving setup does not read.

That second point costs businesses money and nobody puts it on a slide. The model is not the asset. The tuned stack around it is — the cache behaviour, the memory ceilings, the batch sizes, the months of small corrections that turned "it runs" into "it runs reliably". Changing formats means re-earning all of it, for a model I had already calculated would be slower.

Compute the traffic before you start the download

Public comparisons rank models against each other. They do not rank them against your hardware, and your hardware is the constraint you actually have.

It is a three-number calculation: active parameters, bits per weight, and the same pair for today's model. The ratio between those two products is, closely enough, the ratio of the speeds you will get. Everything else — total parameter count, context window, leaderboard position — is marketing until that ratio comes out in your favour.

If your team migrated to a new model this year, ask whether anyone computed this before the migration or after it. If the answer is "we benchmarked it once it was running", you paid for the download, the integration and the re-tuning to learn what arithmetic would have told you for free.

Closing a question is worth more than leaving it open

I did not put this on a list to revisit. The incumbent stays, the reasoning is written down, the question is shut.

An open question about your infrastructure is a recurring tax. It resurfaces in every planning conversation, and each time someone spends an hour half-answering it. Closing it with a number means it reopens only when something real changes: new hardware, a different compression, a shift in what the work demands.

The method's boundary is sharp: it predicts throughput, not answer quality. It cannot tell you whether the newer model writes better summaries, and I never measured that, because I never ran it. What the calculation bought was a priced trade — I knew the speed I would be paying before weighing any quality gain against it. Most migrations run the other way round, and find the price after the switch.

The bottom line

Bigger and newer loses to smaller and already-tuned more often than launch coverage suggests. Speed on your own hardware is arithmetic, not reputation, and the arithmetic is short enough to do before you commit disk, time or a migration plan.

I spent an afternoon not downloading a model. It is one of the better afternoons I have had this year.

For the engineers

Decode on a mixture-of-experts model is memory-bandwidth-bound, not compute-bound. Tokens per second is approximately effective bandwidth divided by bytes read per token, and bytes per token is active parameters times bits per weight over eight. Total parameter count enters only through whether the thing fits.

Incumbent: 32B active at roughly 3.3 bits, about 13.2 GB per token, measured at about 21 tok/s warm on short context. Candidate: 744B total, 40B active, 1M context. The smallest quantization practically available in my serving format landed at 418 to 430 GB — about 3.6 bits per weight — giving roughly 18 GB per token. That is a 1.36x traffic ratio, projecting to about 15 tok/s on identical hardware. A model with fewer total parameters can still be slower if its active count or bits per weight are higher.

True 2-bit-class builds existed at 217 to 239 GB, comfortably resident, but only in a different runtime format at 76 to 82% of reported accuracy — and switching formats forfeits the tuned prompt cache, wired-memory limits and chunk sizes that took months to settle.

The rule: compute active-times-bits for candidate and incumbent before you download either. The ratio is your throughput ratio, and it is accurate enough to decide on.

The shopping trip that started this — and the follow-up where my own configuration turned out to be the problem, not the market — is I Nearly Bought a Capability I Already Owned. And the machine all of this arithmetic is about is the setup from Cloud vs. Local Models in 2026.

If your team is weighing a move to a newer model — or has already migrated and cannot explain why the new one feels slower — a consulting day will price it: I will run the capacity arithmetic against your own hardware, price the switch per request before you pay for it, and put a number on whether the upgrade is one. Book a consulting day or send me an inquiry with your hardware and the model you are eyeing.