Malhar Ujawane

Blog

Explainer·(updated October 31, 2026)

What are LLM benchmarks? MMLU, GPQA, SWE-bench and more

LLM benchmarks are fixed test sets that score models on knowledge, reasoning, coding or agents. What the main ones measure, why they saturate, how to read them.

What are LLM benchmarks? MMLU, GPQA, SWE-bench and more

An LLM benchmark is a fixed, public set of tasks with an automatic way to score answers, used to compare language models on one capability: general knowledge (MMLU), expert reasoning (GPQA), coding (HumanEval, SWE-bench), agent work (τ-bench, Terminal-Bench) or human preference (Chatbot Arena). A benchmark score tells you how a model did on those questions under that setup. Whether it predicts how the model will do on your task is a separate question, and usually the more important one.

Why do LLM benchmarks exist?

Because you can’t compare models by vibes, and you can’t afford to have humans grade every release. A benchmark fixes the questions and the grading so that two labs, or two versions of one model, get the same exam. Launch posts lead with them, leaderboards rank by them, and they’re the first filter when you choose a model, which I treat as one input among several in how to choose an LLM.

The trouble is that every useful benchmark becomes a target. Labs optimize for it, its questions leak into training data, and within a couple of years the frontier scores sit near the ceiling. Most of the history below is that cycle repeating.

What are the main LLM benchmarks?

MMLU (Hendrycks et al., 2020, ICLR 2021) is multiple choice across 57 subjects, from elementary math to US history and law. Its test split has 14,042 questions (cais/mmlu). When it came out, most models were near random chance and the largest GPT-3 was almost 20 points above it.

GPQA (Rein et al., 2023) has 448 multiple-choice questions in biology, physics and chemistry, written by domain experts to be “Google-proof”. People with or pursuing PhDs in the field scored 65%; skilled non-experts with web access and over 30 minutes per question scored 34%; the best GPT-4 baseline scored 39%.

HumanEval (Chen et al., 2021) has 164 hand-written Python problems: write a function from its docstring, graded by unit tests. It introduced the pass@k metric. The original Codex model solved 28.8%.

SWE-bench (Jimenez et al., 2023) moved coding evaluation to real GitHub issues in real repositories, graded by the project’s tests. I explain it, and why its Verified subset was retired, in what SWE-bench is.

Agent benchmarks grade multi-step work in an environment. τ-bench simulates customer-service conversations and checks the database state at the end. Terminal-Bench 2.0 has 89 hard tasks in terminal environments, each with a human-written solution and tests; frontier models and agents scored under 65% when it was published.

Humanity’s Last Exam (Phan et al., 2025) was built because, in its authors’ words, LLMs “now achieve over 90% accuracy on popular benchmarks like MMLU.” It has 2,500 questions across dozens of subjects, written by subject experts, each with an unambiguous, verifiable answer that can’t be quickly found online. The authors describe it as designed to be the final closed-ended academic benchmark of its kind.

Chatbot Arena (Chiang et al., 2024) works differently: people chat with two anonymous models and vote for the better answer, and the votes become a ranking. It had over 240,000 votes when the paper was written. It measures preference on whatever people choose to ask, which is real signal but not the same as correctness.

Why do benchmarks stop being useful?

Three reasons, and each has a well-documented case.

Saturation. Once top models score near the ceiling, the remaining questions are mostly the hardest or the broken ones, and real progress shows up as tiny score changes. Anthropic’s guide to agent evals uses SWE-bench Verified as the example: frontier models went from about 40% to over 80% in a year.

Label errors. Benchmarks are written by people and contain mistakes. Are We Done with MMLU? re-annotated 5,700 MMLU questions and estimated 6.49% of all MMLU questions contain errors; in the Virology subset, 57% of the analyzed questions had errors. A model can’t score 100% on a key that’s wrong, and a model that matches a wrong key may have memorized it.

Contamination. Benchmarks built from public data end up in training data. OpenAI’s February 2026 post on SWE-bench Verified found that frontier models from three labs could reproduce gold patches or problem details verbatim for some tasks, and that exposure helped them pass. Its advice: publish datasets in ways that keep them out of crawls (password protection, for example) and filter training data strictly by canary strings.

How big a score difference is real?

People compare benchmark scores to one decimal place, but every benchmark is a sample, and small samples are noisy. This script computes the 95% margin of error on a 70% score, and the gap two models need before the difference clears that bar, for several published benchmark sizes.

# bench_noise.py: how big a score gap has to be before a benchmark can see it.
# Python 3 standard library only. Sizes are the published question counts.
import math

BENCHMARKS = {"Terminal-Bench 2.0": 89, "GPQA": 448, "SWE-bench Verified": 500,
              "Humanity's Last Exam": 2500, "MMLU (test)": 14042}

def margin(p, n, z=1.96):          # 95% half-width for one model's accuracy
    return z * math.sqrt(p * (1 - p) / n)

def min_gap(p, n, z=1.96):         # smallest difference between two models that clears 95%
    return z * math.sqrt(2 * p * (1 - p) / n)

print(f"{'benchmark':<22}{'items':>7}{'score 70% +/-':>15}{'gap needed':>12}")
for name, n in BENCHMARKS.items():
    print(f"{name:<22}{n:>7}{margin(0.7, n):>14.1%}{min_gap(0.7, n):>12.1%}")

# Label noise: if 6.49% of answers are wrong (MMLU-Redux's estimate for MMLU),
# a model that is always "right" by the key cannot score above this ceiling.
print(f"\nceiling with 6.49% bad labels: {1 - 0.0649:.1%}")

Output (Python 3.13.11):

benchmark               items  score 70% +/-  gap needed
Terminal-Bench 2.0         89          9.5%       13.5%
GPQA                      448          4.2%        6.0%
SWE-bench Verified        500          4.0%        5.7%
Humanity's Last Exam     2500          1.8%        2.5%
MMLU (test)             14042          0.8%        1.1%

ceiling with 6.49% bad labels: 93.5%

This is a rough model. It treats questions as independent coin flips and the two models as independent samples; a paired comparison on the same questions is tighter, and repeated runs add their own variance. But the order of magnitude holds. On an 89-task benchmark, a 5-point gap between two models is noise. On GPQA, a 3-point gap is noise. Only the large benchmarks can separate models a point or two apart, and those are the ones nearest saturation.

The last line is a simplification too, since a model that memorized the wrong keys can beat it, but the point stands: near the top of MMLU, the remaining points measure the answer key as much as the model.

How should you read a benchmark claim?

Ask five questions before trusting a number in a launch post.

  1. Which version and split? SWE-bench, Lite, Verified and Pro are different exams. So are MMLU and its cleaned variants.
  2. What setup? Few-shot or zero-shot, with or without reasoning, which agent harness, how many attempts. pass@1 and best-of-N aren’t comparable.
  3. Who ran it? Vendor-reported numbers use the vendor’s own scaffold and settings. Independent reruns often differ.
  4. Is the gap bigger than the noise? Use the table above as a sanity check.
  5. Could the model have seen it? The older and more public the benchmark, the more likely.

From an engineering seat, I read benchmarks the way I’d read a vendor’s performance numbers for a database: useful for ruling things out, rarely enough to decide between finalists.

What I would do

  • Use benchmarks to shortlist, not to choose. Pick the two or three relevant to your task type (coding, agentic tool use, long-context retrieval), drop models far behind, then stop reading leaderboards.
  • Weight recent, hard, held-out benchmarks. Newer sets like Humanity’s Last Exam, SWE-Bench Pro’s commercial set or Terminal-Bench 2.0 still have room at the top and less time to leak.
  • Build a small private benchmark. Fifty real tasks from your product, graded by code where possible and by a calibrated LLM judge where not. It can’t be contaminated by a public crawl, and it measures what you ship. The method is in how to evaluate AI agents.
  • Report error bars, even rough ones, whenever you compare two models internally.
  • Track cost and latency next to accuracy. A model three points ahead at five times the price per task is a different decision from what the leaderboard implies.

FAQ

What is the most important LLM benchmark?

There isn’t one. Pick by task: GPQA or Humanity’s Last Exam for expert reasoning, SWE-bench Pro or Terminal-Bench for coding agents, τ-bench for tool-using conversational agents, Chatbot Arena for general chat preference. Your own task set beats all of them for your product.

What does MMLU measure?

Multiple-choice knowledge and problem solving across 57 subjects, from elementary math to law and medicine. It was hard when released in 2020; frontier models now score above 90%, so it no longer separates top models well.

What is benchmark contamination?

When benchmark questions or answers appear in a model’s training data, so the model can recall rather than solve them. It inflates scores. OpenAI stopped reporting SWE-bench Verified in 2026 partly because models could reproduce its reference fixes.

Are LLM leaderboards reliable?

They’re useful for a rough ordering. Check the benchmark version, the evaluation setup and who ran it, and treat small gaps as ties. Preference leaderboards like Chatbot Arena measure what users like, which is related to but different from accuracy.

Sources

Get new posts in your inbox

© 2026 Malhar Ujawane. Built with Astro.