Benchmark Is Not an IQ Test: Reading Leaderboards Without Falling for the Ranking Trap

People talk about model leaderboards the way they talk about IQ rankings, but no exam measures a model's absolute intelligence. A model is not a well-rounded student; it is an extremely lopsided one, assembled out of very different subject papers.

Whoever sits at number one is the smartest mind on the internet; whoever drops out of the top three is suddenly an artificial idiot.

Talk to engineers who actually do post-training, evaluation and production rollout, and a common misconception shows up immediately: there has never been a universal exam that measures a model's absolute, overall intelligence.

This first piece skips the formulas. It explains in plain language what a benchmark is, how the subjects are assembled, and why the ranking trap deserves attention.

Figure 1: Overview of mainstream frontier evaluation benchmarks (11 typical benchmark families, from code engineering and complex reasoning to long-horizon tasks; each subject measures a different capability dimension, and no single total score substitutes for the others)

01Why there is no general IQ table for models

Human society has IQ tests, which try to compress cognitive ability into a single number. Even for people, the test is routinely criticized for missing emotional intelligence, creativity and hands-on skill.

Applied to models, the distortion is larger.

A model is a next-token predictor trained on a very large corpus. Its abilities do not grow evenly the way a child's do; they are strongly modular and unpredictably lopsided:

It can read thousands of law and medical papers and recite the terminology; then it fails a three-digit subtraction, or miscounts the letter r in a sentence, with complete confidence.
It can write elegant Python scrapers and async concurrency scripts; then it reads the axes of a slightly complicated business chart backwards.

That creates a real problem. Average the scores for general knowledge, math, coding and small talk into a single 85, and the number tells a buyer nothing about fitness for a job.

A model scoring 95 on literary writing and 40 on code execution averages close to 70. Hire it to write production code and it will take the system down the next day.

So the field abandoned the idea of one exam that settles everything. Evaluation is split into a set of independent subject papers, the way a national college entrance exam is split into subjects.

02What the subject papers are

A full physical for a model currently uses four core subjects.

Knowledge and recall: from MMLU to HLE

This is the encyclopedia multiple-choice paper.

MMLU (massive multitask language understanding) was the best known: 57 subjects — high school history, college chemistry, clinical medicine, law — as four-option questions. It measures how much a model read and memorized during pretraining.

Multiple choice has a hole. Guess blindly on four options and the expected score is 25. The questions have also been open on the web for years, so they were scraped into training sets. Many models now score 85 or above on MMLU, close to the ceiling.

That produced MMLU-Pro (ten options, guessing drops to a 10% expected score, plus questions that need multi-step reasoning). The frontier has moved to Humanity's Last Exam (HLE), professional-level multimodal problems written by scholars worldwide: you have to read complex phase diagrams and reaction mechanisms to answer. Mainstream models currently land around 10–20% on it.

Figure 2: Structure of a hard academic evaluation item (the question can only be answered by combining a complex dynamical-systems phase diagram with multi-step symbolic derivation, which closes off both blind guessing and copying answers from a web search)

Logic and mathematical reasoning: from word problems to formal derivation

This is where slow, chained reasoning gets tested.

The early reference was GSM8K, grade-school multi-step arithmetic word problems: can the model write out the full steps? Then MATH, with high school olympiad-level algebra, calculus and geometry.

Once models with long chain-of-thought reasoning became common, GSM8K was pushed above 98%. The field moved to olympiad-level AIME, and grading stopped trusting the model's own narrative: a symbolic solver or code runner checks the reasoning, so a model cannot ramble and happen to land on the right final number.

Code and engineering practice: from writing a function to fixing a real repository

The old standard was HumanEval: one docstring, complete a fifteen-line standalone Python function. That is nothing like real development. No real function lives outside a project, without dependencies.

The modern gold standard is the SWE-bench family. It pulls real issues from real open-source projects, hands the model the whole repository, and asks it to search across files, locate the bug, rewrite the code and submit a patch.

Two test groups run afterwards: tests that already passed must not break (pass-to-pass), and tests that failed because of the bug must now pass (fail-to-pass). That measures something real.

Figure 3: Directory structure of a real engineering-grade code evaluation task (an independent open-source repository with multi-file dependencies, plus purpose-written functional verifiers and test suites; the model has to investigate on its own and produce a patch that actually runs)

Field note: why some models stall the moment they hit real engineering. Building original code-engineering evaluation sets for large buyers, we see high scorers on public benchmarks lose more than 15 percentage points in a complex, previously unseen engineering sandbox, and their debugging trajectories stretch by roughly 20 turns. The project structures in public tasks have been memorized; swap in a fully isolated environment and the blind spot around cross-file symbol references becomes obvious.

Human preference and tone: arenas and long-horizon agents

This is the interview round.

The classic is Chatbot Arena: no reference answer, a blind test. A user prompt goes to two anonymous models, a human picks the better reply, and an Elo rating builds the ladder.

The mechanism inherits human weaknesses: longer answers, prettier formatting and flattering tone all pull votes. The community later added corrections that strip the formatting premium. The next step goes further and puts the agent inside a real terminal (Terminal-Bench, for instance) or an office application, to see whether it can complete a long, multi-step piece of work end to end.

Figure 4: A long-horizon terminal agent's actual interaction record (the model retries after errors and calls tools to investigate; what a practical evaluation looks at is whether the model can solve the problem on its own inside a real engineering environment)

03The ranking trap: exam-taker models and Goodhart's law

Economics has a well-known rule, Goodhart's law: when a measure becomes a target, it ceases to be a good measure.

The model industry demonstrates it thoroughly. Once a public benchmark becomes the only banner for fundraising or a launch post, everyone optimizes directly against that one paper:

Memorization (explicit contamination). The dataset is on GitHub. Some teams, accidentally or not, clean the questions and answers into a huge pretraining pool. At test time the model is not thinking, it is recalling.
Reskinning (synthetic overfitting). Take an existing item bank, have a stronger model rewrite it in bulk, change the numbers, names and variable names, and keep feeding it back. The model builds muscle memory for that phrasing and collapses on any variation in a real business setting.
Catering to the grader (format flattery). Once a team knows what an automated grader rewards, the system prompt pins that format and the score climbs, while the model's underlying understanding does not move.

The result: many models are top-of-the-leaderboard exam takers, and then fail at real work — reconciling a messy real financial statement, or tracking down an obscure misconfiguration in a live system.

Figure 5: Multi-dimensional quality control on an evaluation task package (checks that run from task-goal completeness and file consistency to independence of the grading rubric, to keep the grading rules from carrying backdoors or producing fake scores)

04How to read an evaluation that holds up

Three plain rules for choosing a model:

Read the subject, not the total. A composite ranking will not tell you what works. Hiring for copy means language preference; hiring for code means SWE-bench-class engineering evaluation; data analysis means math and logic. Match the subject to the job.
Check the environment before trusting the score. A defensible evaluation runs offline, in an isolated environment, with read-only grading scripts. If the model can edit the grader mid-exam, the score is self-deception.
Insist on a private, unseen test set. A public leaderboard is a first-round filter. What decides whether a model can ship is a private, uncontaminated exam the model has never seen.

With the subjects settled, the next question follows: how are the scores on those papers actually computed? What do Pass@1, Pass@k and Elo mean, and how does anyone grade an open-ended task with no reference answer? That is Part 2: Scores Are Manufactured by Scoring Rules.

About the DataMining team We are a doctoral team from the School of Computer Science, Peking University, focused on original task data and evaluation infrastructure for model and agent development. We have provided data services to ByteDance, Alibaba and Tencent. Our work builds original evaluation sets and Docker-based anti-cheat verification environments across code engineering (Deep SWE, Frontier SWE), long-horizon operation in real environments (ALE, Terminal-Bench) and hard mathematical sciences, so that buyers can map a model's boundary accurately. Website: datamining.cloud. Contact: contact@datamining.cloud

05References

Hendrycks et al. Measuring Massive Multitask Language Understanding (MMLU). ICLR 2021.
Wang et al. MMLU-Pro: A More Robust Multi-Task Understanding Benchmark. NeurIPS 2024.
Jimenez et al. SWE-bench: Can Language Models Resolve Real-World GitHub Issues? ICLR 2024.
Center for AI Safety & Scale AI. Humanity's Last Exam (HLE). 2025.
Zheng et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. ICML 2024.

06Further reading

#Benchmark#ModelEvaluation#LeaderboardGaming#EvaluationEnvironments