AI benchmark

How an AI benchmark works

At its core, a benchmark consists of three parts: a data set of tasks, a scoring rule and a protocol that specifies how the model receives the tasks. For a knowledge test such as MMLU, this means around 14,000 multiple-choice questions from 57 subject areas, the score is the proportion of correct answers, and the protocol specifies whether the model sees examples (few-shot) or not (zero-shot). For a coding benchmark such as SWE-bench, the task is a real GitHub issue, scoring is done via the project's test suite, and the protocol determines which tools and how many attempts the model gets.

The result is a number: 88.7 per cent on MMLU, 72 per cent on SWE-bench Verified, an Elo score of 1,400 in an arena. This number is only comparable within the same benchmark and the same protocol. Two models measured on the same test with different tool access or different thinking time cannot be compared directly, even if both numbers are in the same table.

The most important benchmark families

Why benchmarks age: saturation and contamination

Two effects ensure that every benchmark has a use-by date.

Saturation occurs when the best models are close to 100 per cent. The test then no longer separates them, and the remaining points are often errors in the data set itself rather than weaknesses of the model. MMLU was published in 2020, when the best models reached around 44 per cent; a few years later, top models were at almost 90 per cent, and analyses found faulty or ambiguous questions. GSM8K, introduced in 2021 as a demanding primary school maths test, is practically solved today.

Contamination means that test tasks or their solutions were contained in the model's training material. Since language models are trained on large parts of the public internet and benchmark data sets are publicly available on GitHub or Hugging Face, this can hardly be avoided completely. A contaminated model achieves a high score without actually having the capability. With coding benchmarks whose tasks come from public repositories, a model may have seen the actual solution to an issue and simply reproduce it.

The state of research on these problems is well documented; the Wikipedia article on language model benchmarks gives an overview of the concepts and criticisms. Benchmark providers have responded with three strategies: private test sets that are never published; regularly renewed tasks (such as LiveCodeBench, which only scores tasks published after a model's training cut-off); and interactive formats in which there is no fixed solution to memorise, because the model first has to explore an environment.

Measurement protocol: thinking time, tools, attempts

Since models have been able to "think" before answering, the protocol has become an influencing factor in its own right. The same model version achieves a different score on the same test with a low thinking level than with the maximum, and the difference can be in double digits. Then there is tool access (may the model execute code or search the web?), the number of attempts (one run or the best of eight?) and the compute budget in US dollars per task. Reputable rankings state these parameters, for example as "pass@1" for one attempt or as a cost figure per test run. Anyone comparing two numbers should first check whether they were produced under the same protocol. A score without a protocol statement is almost worthless for a decision.

Why this matters for companies

Anyone at a mid-sized company choosing a model for a process makes a decision with follow-up costs: token prices, integration effort, training and possibly a change of provider. Benchmark tables are the obvious starting point, but they mislead in three ways.

• The benchmark does not measure your task. A model that leads in maths olympiads is not automatically better at extracting German supplier invoices or writing product texts in the brand voice. For most business tasks, there is no public benchmark.
• Vendor figures and independent figures differ. Manufacturers measure with the most favourable configuration for their model: maximum thinking time, their own tool access, sometimes several attempts with the best one selected. Independent bodies measure with a uniform protocol and budget. The difference is often five to ten percentage points.
• Costs are missing from the table. A model that is two points better and costs five times as much is the wrong choice for a batch process with millions of calls. Metrics such as price per solved task or cost per percentage point are more meaningful for purchasing decisions than the raw score.

The practical consequence: public benchmarks are suitable for pre-selecting two to four candidates. The decision should be made by your own test with real data from the process, with a fixed scoring rule and a comparison of costs per task. This in-house evaluation set is at the same time the basis for later model changes, because it measures every new version against the same cases.

A real-world example: SWE-bench and its verification

SWE-bench was published in 2023 by researchers at Princeton University. The data set consists of more than 2,000 real GitHub issues from twelve popular Python projects. A model receives the problem description and the repository and has to produce a patch that makes the associated tests pass. This was an advance on HumanEval, which only tested small, isolated functions: SWE-bench checks whether a model can find its way around unfamiliar code.

It soon became apparent that many tasks were unsolvable or underspecified, because the issue description was insufficient or the tests checked more than was stated in the issue. In 2024, OpenAI therefore had software developers review every task together with the authors and published SWE-bench Verified, a cleaned-up subset of 500 tasks. Within around two years, the best results rose from under 5 per cent to over 70 per cent. Then the pattern repeated itself: the verified subset also came under suspicion of saturation and contamination, because the solutions to the issues are publicly available in the repositories. The community moved to harder and fresher variants (SWE-bench Pro, Multi-SWE-bench, Terminal-Bench), and several providers no longer report the original score in their model cards.

The lesson from this story applies to every benchmark: a good test has a lifespan of a few years, and a score of over 90 per cent says more about the age of the test than about the model.

Distinction: what a benchmark is not

A benchmark is not an evaluation. Evaluation is the umbrella term for any systematic testing of a model, including your own tests, red teaming and user studies. A benchmark is a standardised, publicly comparable form of it.

A leaderboard is not a benchmark. Rankings such as those by Artificial Analysis or LMArena combine several benchmarks or user votes into an index. The weighting of the individual tests is an editorial decision by the operator and changes the order.

An Elo score is not a percentage. Arena ratings result from pairwise comparisons and are relative: they say how often a model wins against others, not how often it is right. Preference benchmarks also reward style and detail, which is not always a good indicator of factual accuracy.

"Human-level" is a moving target. Many benchmarks state a human reference, but the comparison group ranges from crowd workers to experts with doctorates. A model that reaches "expert level" on GPQA is not therefore an expert in the profession.

Outlook

The benchmark landscape is shifting in three directions. First, towards agentic, interactive tests in which the model acts in an environment, uses tools and pursues a goal over many steps; here, contamination is difficult and the proximity to real work is great. Second, towards economically grounded tests in which real professional work results are assessed by experts instead of counting multiple-choice questions. Third, towards efficiency measures: in addition to the result, costs, latency and token consumption per task are reported, because models with adjustable thinking time can pass the same test at very different prices. For companies, it is therefore becoming more important to formulate their own task as a benchmark than to follow the public rankings.

Frequently asked questions

Which AI benchmark is the most important?
There isn't one. For coding, agentic tests such as SWE-bench variants and Terminal-Bench are relevant; for specialist knowledge, GPQA and Humanity's Last Exam; for the everyday quality of answers, arena ratings. For a purchasing decision in a company, the most important benchmark is your own, with real data from the process.

Why do manufacturers report higher scores than independent testers?
Because they measure with the best configuration for their model: maximum thinking time, their own tools, sometimes several attempts. Independent bodies use a uniform protocol for all models. Both numbers can be correct; they are just not comparable.

What does it mean when a benchmark is "saturated"?
That the best models are so close to the maximum that the test no longer distinguishes between them. The remaining errors are then often errors in the test data set. A saturated benchmark is still useful for testing weak models, but no longer for comparing the top.

How do I build my own benchmark for my use case?
Collect 50 to 200 real cases from the process with a clearly verifiable target output, define a scoring rule (exact match, checking by a second model against a list of criteria, or expert assessment), and measure accuracy, latency and cost per case for each candidate model. Keep test cases separate from anything that flows into prompts or fine-tuning.

Can a model "memorise" benchmarks?
Yes, that is called contamination. Public test questions end up in the training material, and the model reproduces the solution without understanding the task. Reputable benchmark operators therefore keep parts of their tasks secret or renew them regularly.