How AI benchmarks work

A benchmark is a set of questions with known answers. This one is multiple choice: four options, one right.

The model’s choice comes from probabilities, like every token it writes. Here they fall on four letters.

It answers B. Many tests let a model reason first, then read only its final answer.

The answer is checked against the key: one point if it matches, none if it does not. No judgement is needed, which makes multiple choice cheap to grade.

GPQA Diamond has 198 graduate-level questions in biology, chemistry and physics, written by experts to be hard to look up.

An illustrative model answers every question and gets 159 right, one question at a time.

159 of 198 is 80.3%. With four choices each, guessing alone scores about 25%: the floor is not zero.

The best result on GPQA Diamond today is 95.8%, by GPT-6 Astra (max). When the best models crowd near 100%, a test can no longer tell them apart.

Think of the 198 questions as a sample of every question the test could have asked. Another sample of the same kind would give a slightly different score.

With 198 questions and a score of 80%, the standard error is about 2.8 points: 95% of such tests would land between 74.8% and 85.8%.

Epoch AI publishes a standard error with each result. On GPQA Diamond, 8 of the top eight reach into the leader’s range: too close to rank with confidence.

Multiple choice is graded by matching one letter to the key.

Maths tests often want a single number and accept only that number. In the AIME exam, every answer is a whole number from 0 to 999, so guessing almost never works.

SWE-bench gives a model a real issue from an open-source Python project. The model changes the code, and scores only if the project’s own tests pass.

Some qualities have no answer key. Arena shows a person two anonymous answers to their own question, and they pick the better one.

Each vote says only that one model beat another, once.

A statistical model, Bradley–Terry, finds the scores that best explain every vote. The further apart two scores, the more often the higher one is expected to win.

Arena publishes a 95% range with each score. Among its text leaders, 1 of the top eight reach into the leader’s.

A test’s questions are meant to stay out of a model’s training data.

But a published test lives on the web, and training data is gathered from the web. A copy can slip in: this is called contamination.

A model that has seen a question can answer it from memory. Its score on those questions can rise without any more skill.

LiveBench adds new questions every month from recent sources. Other tests keep a set private, and researchers search training data for copies.

An easy test tells weaker models apart but leaves the best tied near 100%. A hard test does the opposite.

Fit the tests together: each gets a difficulty, each model a capability, and a curve links the two. Tests at every level then place all the models on one scale.

The Epoch Capabilities Index combines more than fifty benchmarks this way. Its scale is set so that Claude 3.5 Sonnet scores 130 and GPT-5 150: compare its numbers, don’t read them alone. Today Claude Opus 5.5 leads, and 8 of the top eight reach into its range.