The Benchmark That Lied About Production

One number in my server's configuration has made a fool of me twice, in opposite directions. The number itself never moved. Both times I was wrong with clean evidence in my hand.

The machine is an M3 Ultra Mac Studio with 512 GB of unified memory, serving language models all day for my own work and for clients. One setting controls how big a bite it takes out of a request while reading it. Bigger bites, in theory, mean faster reading.

So I measured it properly. Six values, clean runs, one at a time, nothing else touching the box. The winner was unambiguous: 210 tokens per second against 161 at the setting we had been running — a 29% gain, reproducible, no hand-waving. I deployed it. It took the server down.

The benchmark was honest. It was also useless.

Here is the curve I got, in tokens processed per second, sweeping the bite size from small to large: 161, then 200, then 210, then 207, then 118, then 116.

A textbook result: performance climbs, peaks, goes flat, then falls off a cliff. The two largest values came in at 73% and 72% of baseline — worse than the setting I started with. The optimum sat in the middle, 29% above the incumbent.

I translated it into the only units that matter to anyone paying for this. On a 10,000-token document — a long contract, a research brief — the reading stage dropped from 62 seconds to 48. On 20,000 tokens, from 124 to 96. That is the kind of number that ends a meeting early.

Every one of those figures is real. Every one of them is also irrelevant, because of one thing I never varied: I sent the requests one at a time.

One request at a time is a demo, not a test

Production is not one request. My server is configured to hold up to 25 requests in that reading stage simultaneously, because that is what real traffic looks like — overlapping, ragged, arriving while the previous one is still mid-flight.

Each of those 25 requests needs its own temporary working memory while it takes its bite. Make the bite bigger and every one of those scratch spaces grows at the same time. A benchmark that sends one request at a time never has two of them alive at once. It is structurally blind to the problem.

Under real concurrent load, the new setting drove memory use to 396 GB of the machine's 512 GB and the graphics allocator gave up — an out-of-memory failure, not a slowdown. The server did not get slower. It stopped.

The lesson for businesses: your load test has to contain your load. A test run under the wrong conditions produces a confident, well-evidenced, wrong answer — and one that is hard to argue with, because the numbers themselves are correct.

I gave up four percentage points to keep the server standing

I rolled back one step below the peak. That setting reads 200 tokens per second instead of 210 — about 24% faster than where I started rather than 29%, with memory headroom that survives a busy hour.

The five points I "lost" were never available. They existed only in a room with one user in it. The optimum you find in isolation and the setting that survives real traffic are two different numbers, and the gap between them does not arrive as a slowdown you can monitor your way out of. It arrives as an outage, at your busiest moment.

If a benchmark chose a performance setting you run in production, ask what the concurrency was while that benchmark ran. If the answer is "one", or "nobody recorded it", the setting is untested. It may still be fine. You do not know yet.

A setting can be correct while your reason for it is wrong

The same number has a second story, and this one catches careful people rather than sloppy ones.

That value had been inherited. It was chosen for a much larger model on the same machine, back when memory was the binding constraint and small bites were an act of caution. When a very different model took over — different architecture, far more memory to spare — the reasonable worry was that the old caution was now throttling it. The obvious move was to raise it.

I swept it instead: 1,578, then 1,783, then 1,868, then 1,822, then 1,733, then 1,592 tokens per second across six sizes. A clean inverted U — and the peak sat exactly on the inherited value. Moving in either direction cost roughly 15% of throughput. The "obvious" correction would have made the system measurably worse.

The setting was right. The reason for it was extinct. It had been chosen to conserve memory; it is now optimal because most of this model's layers use an attention design that gets no faster from bigger bites. Right answer, wrong argument, pure coincidence.

For a business, that is the more expensive failure. Nobody audits a setting that appears to be working. It stays accidentally correct until the coincidence expires — a model swap, a hardware upgrade, a version bump — and by then the reasoning that would have caught it is long retired.

Test the failure you are afraid of, not the one that is easy to stage

Both stories reduce to one mistake: trusting a measurement taken under conditions that do not match reality, or an assumption that quietly stopped being true.

One caveat, and it matters. These numbers came from one machine and one class of workload; your hardware will produce different figures. What transfers is the shape, not the values: performance curves have a cliff on the far side, the ceiling under concurrency sits below the ceiling in isolation, and an inherited setting is a coincidence until you measure it.

The bottom line

A benchmark measures what you staged, not what you will get. Single-request tests report a ceiling no real user will ever stand under, and they fail silently in the most flattering direction.

I run 24% faster today, not 29%. Twenty-four percent that stays up beats twenty-nine percent that falls over on your busiest afternoon.

For the engineers

The setting is the prefill chunk size — how many prompt tokens go through the forward pass per step. Larger chunks improve arithmetic intensity until the matmuls saturate the accelerator; past that, intermediate activations grow, you go bandwidth-bound, and throughput collapses. Isolated sweep at 1024 through 32768: 161 / 200 / 210 / 207 / 118 / 116 tok/s — peak at 4096, flat at 8192, hard regression below baseline from 16384 up.

The crash: the server permits 25 concurrent prefill slots. Peak transient allocation scales with chunk size times in-flight requests, and a serial benchmark exercises exactly one of them. Under overlapping production traffic at 4096, resident memory reached 396 GB of 512 GB unified and the GPU allocator failed outright. Rolled back to 2048: 200 tok/s, roughly 24% over baseline, with headroom for a burst.

The second sweep, on a model whose majority of layers use recurrent attention, ran 512 through 16384 at 1,578 / 1,783 / 1,868 / 1,822 / 1,733 / 1,592 tok/s — optimum at 2048, roughly 15% penalty either way. Recurrent layers do not parallelize across a wider chunk the way full attention does, so the larger chunk buys no compute efficiency while still costing bandwidth.

Practical rule: sweep at your production concurrency limit, not at one, and record peak memory alongside throughput. A throughput-only sweep will happily hand you an optimum you cannot deploy.

This is the hardware-flavoured version of a lesson A Test Is Not a Load tells about capacity planning generally: a test that doesn't contain your load is a demo. And the machine in question is exactly the setup I describe in Cloud vs. Local Models in 2026 — the trade-offs of running models on your own desk in the Gulf.

If your AI or data infrastructure was tuned by benchmarks run one request at a time — or you are about to sign off on a capacity plan built from vendor throughput figures — that is a consulting day with an obvious agenda: I will re-run the numbers under your real concurrency, show you where the ceiling actually sits, and leave you with headroom sized for your worst hour instead of your best one. Book a consulting day or send me an inquiry first if you'd rather put the question to me in writing.