Benchmarks

As soon as more than one model is on the table, the question arrives: which one is better? Benchmarks promise an answer in a single number. That's convenient - and that's exactly where the trouble starts.

What a benchmark is

A benchmark is a fixed collection of tasks with known solutions. You let the model answer and count the hits. A few names you'll see everywhere:

• MMLU - knowledge questions across school subjects and professions, multiple choice.
• HumanEval - small programming tasks; counts whether the generated code passes the tests.
• GPQA - expert questions that even people with degrees fail at.

The result is a percentage, and percentages sort beautifully into tables. That's why every model announcement carries one.

Why benchmarks lie

Three reasons to distrust the numbers. First, contamination: the test questions have been on the internet for years - and therefore often in the training data. The model didn't pass the exam, it knew it by heart. Second, benchmarketing: whoever releases a model picks the benchmarks it shines on. Third - and most importantly: a benchmark measures its tasks, not yours. A model can win MMLU and still fail at your support emails.

Arenas: people instead of task lists

Comparison arenas like the Chatbot Arena take a different route: people ask a question, get two anonymous answers and pick the better one. That produces a ranking using the same system as chess (Elo). It's harder to game - but it measures what people like, not what's correct. Polite, elaborate answers win even when they're wrong.

Your own benchmark beats every leaderboard

The most honest number is the one you produce yourself: collect twenty real tasks from your agent's daily work - each with its expected answer. Let every candidate answer, count the hits. A mini eval like that is built in an afternoon, reruns on every model switch, and answers the only question that matters: does my agent work with this?

Benchmarks are maps, not the territory: good for sorting the flood of models and shortlisting candidates - unsuitable as the final word.

The final word is an eval on your own tasks. That's no dark art but the same loop your harness already masters: task in, check the answer, count the hits. Keep your twenty test cases alive and you can read every model headline with perfect calm.