Benchmarks

Tasks done in a shell: build, debug, configure, train. The benchmark for coding agents in a terminal.

The length of task, in human hours, that a model completes half the time. Doubling about every seven months.

Harder grid puzzles. Frontier models scored in single digits at launch while people solved most of them.

3,000 expert questions across 100 subjects. Built to be the last academic benchmark.

Research-level maths problems written by mathematicians, kept private. Under 2% at launch; o3 reported 25% a month later.

4,326 short fact questions with one right answer each. Measures how often a model makes things up.

500 SWE-bench tasks checked by people so the tests are fair. The number labs quote for coding agents.

Agents serving simulated customers in retail and airline settings, with rules to follow and a database to change.

MMLU with ten choices, harder questions and the errors removed.

New questions every month from recent sources, so training data cannot contain the answers.

369 tasks in real Ubuntu, Windows and macOS desktops for computer-use agents.

11,500 college-level questions with images, across 30 subjects. The multimodal MMLU.

Google-proof PhD-level science questions. Frontier models passed experts in 2025.

466 questions that need tools, browsing and several steps. People score 92%; the first agents scored 15%.

Fix real GitHub issues in real repositories. The coding-agent benchmark. From 2% in 2023 to above 70% on Verified in 2025.

812 tasks on self-hosted websites for browsing agents. 14% at launch.

Hugging Face's public ranking of open models on a fixed set of evals. Retired in 2025 after the evals saturated.

80 multi-turn questions judged by GPT-4. Showed that a strong model can grade chat quality about as well as people.

Anonymous pairwise votes from millions of users, ranked by Elo. The leaderboard labs watch.

Stanford's framework that scores models on many scenarios and many metrics at once, not one number.

The 23 BIG-bench tasks where models still trailed people. Chain of thought closed most of the gap.

204 tasks contributed by 450 authors. BIG-bench Hard is the subset that still matters.

Grade-school maths word problems. The reasoning benchmark of 2022 and 2023.

817 questions people get wrong because of false beliefs. Bigger models were less truthful at first.

974 short Python tasks written by crowd workers. The companion to HumanEval.

164 Python functions from docstrings. The code benchmark until SWE-bench.

12,500 competition problems with step-by-step solutions. 7% in 2021, above 90% for reasoning models in 2024.

57 subjects, multiple choice. GPT-4 scored 86% in 2023. Saturated by 2024.

Chollet's grid puzzles that need no knowledge, only abstraction. The o3 model reached 87% in December 2024. ARC-AGI-2 followed in 2025.

44,000 Winograd-style pronoun problems, filtered to remove the shortcuts models used.

Harder successor to GLUE. Saturated by 2021.

Pick the sentence that follows. Adversarially filtered so models scored 48% while people scored 95%. Saturated by 2023.

Industry benchmark for training and inference speed on fixed workloads. The scoreboard for chips and clusters.

Nine language understanding tasks. BERT beat the human baseline in 2019.

57 Atari 2600 games as a reinforcement learning benchmark. DQN played them from pixels in 2013; Agent57 beat the human baseline on all of them in 2020.

Pronoun puzzles that need common sense, proposed as a replacement for the Turing test. GPT-3 reached 89% in 2020.

The yearly image classification contest, 2010 to 2017. AlexNet in 2012, ResNet in 2015. Machines passed human accuracy in 2015.

The yearly translation contest. Statistical systems until 2015, neural ones after. Google's 2016 system was judged on it.

N-gram overlap with reference translations. The machine translation score for twenty years, and still reported.