Español

Benchmarks Flatter New Models

Every few weeks a new model — often an open-weight one — arrives with a wall of charts showing it matched or beat the frontier at a fraction of the price. The charts aren't fake. But they flatter. On easy, clean, well-scoped tasks, almost everything looks tied, because public benchmarks saturate: the ceiling is low enough that a dozen models bunch up at the top. The real gaps only open on hard, messy, open-ended, long-horizon work — and that's exactly the work the headline chart doesn't test.

Where the charts mislead

Two chart types deserve extra suspicion. The first is the vendor-supplied chart — the model's own lab showing it head-to-head with the leaders on a curated benchmark set. It's real signal, but the vendor chose the axes. The second is the subjective "arena" vote: blind A/B tests where you're shown two outputs — say, "which landing page do you prefer?" — and pick one, Pepsi-challenge style. Aggregated over many votes it's genuine signal, but it's very subjective and easy to cherry-pick. A model can top a front-end arena and still be the wrong tool for your actual job.

The head-to-head the charts hide

Here's a concrete run from a reviewer testing an open model against the frontier on the same creative front-end task. On the scoreboard, the open model roughly matched the leader — the output was solid, competitive on quality. Then the real numbers landed. It took about 90 minutes versus roughly 17 for the frontier leader. It burned about 21 million tokens — a token hog — and, once you accounted for real usage, it cost more than the cheap per-token sticker implied. On a separate hallucination-style "omniscience" benchmark of hard, gray-area questions, the frontier model led clearly while the open model trailed. Same scoreboard, wildly different experience.

The metrics benchmarks bury

Leaderboards optimize for a single quality number. The axes that actually decide whether a model is usable rarely make the headline:

• Total cost per task, not price per million tokens. A cheap per-token rate on a model that spends three times the tokens is not cheap. Independent trackers make this explicit — Artificial Analysis computes cost per task from input, cache-hit, cache-write, reasoning, and answer token prices divided by task count, rather than a sticker rate, precisely because token efficiency varies wildly between models.
• Wall-clock speed. A model that's "just as good" but takes five times as long changes how you work. Ninety minutes per task means you run one experiment, not four.
• Focus across a long loop. Can it hold the thread through 10, 20 tool calls — read the repo, plan, edit twenty files, run tests, read failures, fix — without drifting around step 10? That's the axis a coding agent lives or dies on, and no single-shot benchmark measures it.

Run your own hardest task

The fix is simple and cheap. Ignore the leaderboard for a moment. Take your own hardest, most realistic task — a real bug in a real repo, not a toy prompt — and run it on both models. Then judge three things: total cost, wall-clock time, and whether the model kept its focus across the whole loop. A benchmark win is a headline; your task is the real test.

What to measure yourself

• Total cost of the finished task, from your actual bill — not the per-token rate.
• Wall-clock time from prompt to done.
• Tokens consumed for the same result — the token-efficiency gap hides inside the cheap sticker.
• Focus across the loop — did it hold the plan through many tool calls, or drift and need rescuing?
• The hard, gray-area cases — nuance and long-horizon reasoning, where the gaps actually live.

Takeaway

Benchmarks flatter new models because easy tasks tie and hard tasks separate — and the charts test the easy ones. Vendor charts and arena votes are signal, not proof. Before you rip out a working setup, run your own hardest task on both and measure cost, speed, and focus. The scoreboard is a headline. Your task is the verdict.

Resources

• Cost Per Task, Not Per Token — Yeda AI Tips #203
• Open Weights Isn't Local, and Isn't Free — Yeda AI Tips #204
• Artificial Analysis — independent model benchmarks