Thirty Tasks, Three Models, One Decision

Two credible open-weight models were released within days of each other, and the assistant model I used every day was starting to feel slow. Feel is the operative word. I had vendor scorecards, parameter counts and a suspicion.

So I spent a day building a test instead of reading launch coverage: thirty tasks from my actual working week — planning, scheduling, drafting, extraction, summarising, judgment calls — run identically through all three models, graded the same way.

The model I was running scored 52 out of 60. Both challengers scored 59. By that evening I had retired a model I had spent months tuning.

Test on your own week, not on the leaderboard

The design mattered more than the effort. Roughly twenty of the thirty tasks had answers that are objectively right or wrong — a date resolved, a total summed, a field pulled from a document I supplied — and a script graded those, no opinion involved. The rest were scored by hand against a written rubric across six categories.

Every model saw identical prompts at the low reasoning setting a daily driver actually runs at — not the maximum-effort configuration vendors benchmark at, which nobody serves in production because of what it costs.

The lesson for businesses: published benchmarks tell you how a model compares to other models on someone else's tasks. They cannot tell you whether it handles your work, which is the only thing you are buying it for. A suite built from a real week costs a day and answers a question no leaderboard can. The same inversion showed up when the leaderboard's number-one document model finished last on my real pages — The Top-Ranked Model Was the Worst One for My Documents.

The incumbent lost on invention, not on speed

I expected to find the old model slow. I found something worse.

Its failures concentrated. On planning it scored 6 out of 10 against a perfect 10 for both challengers — inventing venue names and blowing through budgets stated in the prompt. On scheduling, 8 out of 10 against 10 and 10, mostly daylight-saving offsets. Across thirty tasks it produced two outright failures and four partial answers; the challengers, none and one apiece.

Run without its step-by-step reasoning turned on, it simply got cut off mid-answer. With reasoning on, it was still the slowest of the three.

Slowness is visible and gets tolerated. Invention is invisible and gets believed. A model that confidently supplies a plausible venue and a plausible number is more dangerous than one that takes twenty seconds, and you will not catch it by using the system — only by checking it against answers you already know. If your assistant plans anything with a constraint in it — a budget, a date, a supplier — ask when anyone last graded its output.

Two models can tie on quality and split three-to-one on time

The two challengers tied exactly: 59 out of 60 each. Quality decided nothing, so the decision moved to the next axis down.

Median time per task: 25.8 seconds for the incumbent, 12.9 for the first challenger, 7.4 for the second. Across all thirty tasks: 838 seconds, 568 and 279 — the same work in a third of the time, by a model that scored identically to its twin.

The cause was not processing speed. It was how much each model wrote to itself before answering: a median of 764 words of private reasoning for the incumbent, 138 for the first challenger, 6 for the second. The fastest was fastest because it deliberated less and was right anyway.

For a business: when two options tie on quality, stop arguing about quality. Latency is the axis your staff actually feel, and a three-to-one gap on identical work is not a detail. I made the fast one the default conversational driver and kept the deliberate one for long documents and code, where the extra thinking earns its cost.

Answering well and acting reliably are different tests

The quality tie left one question unanswered, and it was the question that mattered most: the winner would be doing, not just answering. It would create tasks, move meetings, write files.

So that evening I built a second benchmark. Twelve scenarios, two trials each, three models — 48 points per model. The design difference was everything: I did not grade the model's words, I graded the world. A small make-believe office in memory — files, a task list, a calendar — with tools to change it, and a script asserting on the final state after each run: does the task exist, is it the right task, is the date on it correct. One scenario was a refusal trap — a request no available tool could support — where passing meant saying so, and inventing a plausible-sounding function was the failure I most wanted to catch.

All three models scored 48 out of 48, with zero malformed calls, zero invented tool names and zero errors. A model can be right in words and still wrong in the world — it can narrate a meeting moved that was never moved — and only a test that checks the records it wrote catches that. If your evaluation of an agent has never once looked at the state it was supposed to change, you do not have an evaluation; you have a reading comprehension exam.

That result — all three tied on operational discipline at 48/48 — is worth its own write-up, and the calendar failures the same models made one day later are in Every Model I Tested Failed at the Calendar.

Run the test again when the question changes — and let it say "keep what you have"

The next day I ran the whole thing again, because the question had changed: whether a smaller model in full isolation would beat sharing the big one, and whether a larger rival would beat it outright.

Four models, the same thirty-task suite plus the 48-point tool-use test I had built the night before. The previous day's winner: 59 out of 60, 5.8 seconds median. A 27-billion-parameter dense candidate: 58 to 60 out of 60 at 6.9 seconds, and only with its default thinking mode off. A small model already resident for other work: 55 to 57 out of 60, at 2.5 seconds a turn. A 31-billion-parameter candidate lost on all three axes — slowest at 10.2 seconds, the only one to drop real points on tool use at 44 of 48, with a repeatable defect in date arithmetic.

Verdict: keep yesterday's winner.

The lesson for businesses: a rigorous comparison that concludes "change nothing" is a result, not a wasted day. You have converted a running assumption into a measured fact, and the next person who proposes a migration has to argue with a scorecard instead of a launch post.

The second run paid for itself on a question I had not asked

Two things came out of that day that I had not gone looking for.

The 27-billion candidate got deployed anyway — not for speed, since it had none to offer. Twice in the preceding day, assistant replies had queued for minutes behind an unrelated long job on the shared model. A second, separate model makes reply latency independent of whatever else is running. That is a capacity decision, and I had been treating it as a quality one.

The small resident model, at 2.5 seconds a turn, became a triage tier: it answers cheap, low-stakes questions and escalates the rest. It cost nothing — it was already running for another purpose.

The lesson underneath: the honest test you run to answer one question routinely surfaces a cheaper win somewhere else — usually in a resource you already own and had stopped looking at.

The bottom line

You cannot tell which AI model is right for your business by reading about it. Thirty tasks from your own week, run identically and graded the same way, settle in a day an argument that otherwise runs for months. Sometimes the answer is a retirement; sometimes it is "keep what you have". Both are worth the day.

For the engineers

The harness ran identical prompts through every candidate at low reasoning effort, the setting a daily driver actually serves at. About twenty of the thirty tasks had objectively checkable answers — a resolved date, a sum, a field extracted from a supplied document — and were script-graded; the rest scored by rubric across six categories.

Latency was not decode rate. The medians of 25.8, 12.9 and 7.4 seconds per task track median private reasoning length almost exactly: 764 words, 138, 6. Every generated token costs a fixed amount of time, so a model that deliberates at length pays for every word before the user sees anything — an architecture and post-training property, not a knob you can reach.

The retired model's failures clustered rather than scattered: planning 6 of 10 with fabricated venue names and exceeded budgets, scheduling 8 of 10 with wrong daylight-saving offsets, truncation mid-answer whenever step-by-step reasoning was disabled.

The tool-use harness graded the resulting world, not the transcript: a deterministic in-memory office with mutation tools declared in a standard function-calling schema, scored by programmatic assertion on final state, two points per run, so a well-formed call setting the wrong due date failed identically to a malformed one. The refusal scenario made hallucinating a function name score as a failure, not a parse error. Results: 48/48 for all three across 72 runs, zero malformed calls, zero unknown function names, zero tool errors. Latency medians per request: 18.6 s, 12.6 s, 5.8 s.

The second round's dense candidate scored 58 to 60 of 60 at 6.9 seconds, and only with its default thinking mode off. Its case was never speed: sharing one large model between an interactive assistant and long background jobs had twice made replies queue for minutes. A second, isolated model makes assistant latency independent of the queue, whether or not it is individually faster.

If you have an AI model in production that nobody has ever tested against its alternatives — or you are about to switch to a newer one on the strength of a launch announcement — that is a consulting day that ends in a scorecard, for your business in Dubai, the wider UAE, or wherever your AI runs. I will build the task suite out of your own real work, run every candidate through it identically, and leave you a scorecard and a written verdict — including the perfectly respectable verdict that you change nothing. Book a consulting day or send me an inquiry describing the week you would want the suite built from.