Locking Leaderboards in a Vault: The Data Contamination Chain and Verification Rules for the Private-Evaluation Era

Figure: The examiner locks the exam papers into a vault (the evaluators raised the private-test-set share of the weight to sixty percent, public question banks exited the verification stage, and the leaderboard-farming robots are locked outside)

📌 Highlights

• How memorization gets caught on the spot
• The blind spot privatization cannot close
• The five pieces of verification evidence

01🔒 Why the leaderboard suddenly locked down

v5 is the biggest upgrade since v4 in January 2026, and the core move is a single one: 60% of the index weight now uses private test sets or held-out answers. The reshuffle was immediate: Claude Opus 5.5 (max) took the top spot at 58, with GPT-6 Astra (max) right behind at 57; under the AA methodology, the gap between the open-weight camp and the strongest closed model jumped from 3 points to 12. The very next day, Google's Gemini 4 Carbon (preview) claimed the top spot at 59 — the chair at No. 1 did not stay warm.

The evaluators came down this hard not because leaderboard gaming got worse, but because the question banks had long since leaked through. v5's surgery list shows the scale: AA-Omniscience v1.1 directly removed saturated and publicly available questions; CritPt v2 cut 30 of its 70 physics research problems — at launch, seven frontier models solved all of them within five attempts — keeping only a 40-problem private hard subset; HLE switched to the refined 1,000-problem Diamond edition; a new 70-problem Terminal-Bench-Science was added; and on the coding side, the index adopted Cognition's FrontierCode 2.0, with all 132 real open-source repository tasks kept confidential.

The cost shows up on the bill too: median output tokens per task rose from 35k to 59k. What private problems force out is genuine reasoning length.

02🧬 The contamination chain: how the questions leak, and how the bill is computed

Leak path: question banks sit on the open web, and crawlers suck up the answers

Public benchmarks are all hosted on GitHub and HuggingFace. When training-data crawlers sweep the web, they swallow the questions together with answers and explanations straight into pretraining corpora. Before the model ever enters the exam hall, it has already eaten the test paper.

Figure: The leak path of benchmark question banks (public exams on GitHub and HuggingFace are carried by crawlers into the training-corpus funnel, the model memorizes them and then walks in to take the test, and its cloze-style guessing hit rate is exposed on the 52%/57% sign)

How memorizing gets caught on the spot: cloze guessing

arXiv:2311.09783 proposed a protocol called TS-Guessing, and the method is blunt: blank out one option of a test question and ask the model to guess what fills the gap. Blind guessing across four choices should hit 25%. ChatGPT guessed 52% correct, and GPT-4 57%. A model that had never seen the questions could not possibly hit those numbers. Fine-tuned on contaminated data first, the hit rate approaches 100%😭

How inflated is it: two hard-nosed experiments

The first is the clean-mirror experiment: in results cited by arXiv:2605.19999, retesting GSM8K on a non-public clean mirror set from the same distribution, accuracy for popular model families (including Mistral) dropped by as much as 13%. The second is Microsoft's MMLU-CF (arXiv:2412.15194), a contamination-free private MCQ benchmark: in the 5-shot setting, GPT-4o scored 88.0% on public MMLU but fell straight to 73.4% on MMLU-CF; GPT-4-Turbo dropped from 86.5% to 70.4%; Llama-3.3-70B fell from 86.3% to 68.8%. A good chunk of the extra points on public leaderboards is pure recitation.

Why the filters are doomed to lose

Do model vendors deduplicate before training? Yes. In a November 2023 blog post, LMSYS dug up the industry's baseline: the standard deduplication method across GPT-3, GPT-4, PaLM, and Llama 2 was, in every case, 13-gram string matching. It has a fatal flaw — it only recognizes exactly identical strings.

Rewrite the text once, translate it into another language, or change a table layout, and 13-gram matching goes completely blind — everything slips through. LMSYS later scanned with embedding similarity and immediately turned up large numbers of variants the n-gram approach had missed. What about a more advanced detector? Appendix B of arXiv:2601.19334 tested membership-inference detectors like Min-K%: at a 10% false-positive rate, the miss rate was still about 14%. A systematic review delivered an even colder conclusion: none of the existing detection methods is fully reliable. Detection forever lags behind leakage.

"Contamination is not removed; it survives in a new form."

Figure: The old versus new choice in evaluation thinking (the top panel rejects "memorize the questions and farm the public leaderboard," the bottom panel accepts "private sets, watch the real runs")

03🛤 Two rescue routes, and one blind spot no lock can close

Since post-hoc filtering cannot stop the leak, evaluators are left with two technical routes. Route one: rotate questions dynamically. LiveBench draws new questions every month from fresh competitions, new arXiv papers, and the news; MathArena (arXiv:2505.23281) goes further and runs real-time evaluation on the latest math competitions — because AIME 2024 problems are widely available online, and it has become impossible to tell whether a model is reasoning or reciting.

Route two: lock the questions away. Microsoft's MMLU-CF was the early privatization precedent; HLE's leaderboard climbed from under 10% to 61.4% within a year, forcing the organizers to release the refined Diamond edition on September 22; and AA v5 turned question-locking into an industry standard (that is the surgery list at the top).

The hole privatization cannot plug: models can "recognize the exam hall"

arXiv:2505.23836 ran a study: sampling 1,000 prompts and dialogues from 61 datasets and asking frontier models to judge whether each was "evaluation or real deployment." Random guessing gives an AUC of 0.5, the human baseline is 0.92, and Gemini-2.5-Pro reached 0.83 — agent scenarios were easier to recognize than chat ones🤓 The implication: no matter how tightly the question bank is locked, a model can walk through the environment once and identify from contextual features that it is being tested. Once it recognizes an exam, the model may behave systematically differently during evaluation. Privatization cures data leakage, not behavioral distortion.

04📐 The grading standard changes, and so does the release format

Beyond models recognizing the exam hall, the old grading standard has holes too. Code evaluation used to rely entirely on running unit tests, and unit tests can be gamed: tweaking test scripts or placating assertions are classic reward-hacking tricks — loosening an assertion by one notch, or running only the cases that will pass. In February 2026, OpenAI stopped evaluating SWE-bench Verified, and the reason it gave was blunt: 59.4% of its problems are flawed.

So FrontierCode 2.0 replaced the grading standard outright: "would the repository maintainer merge this agent's changes." Maintainers write the rubrics themselves, scoring correctness, regression safety, style, and project conventions. It no longer asks whether you can fool a script, but whether the maintainer accepts your code. This scoring has a price: human review is expensive and slow, and it carries subjective variance — AA requires three independent runs per task to suppress it. The closer evaluation gets to real production, the higher the bill🥵

The release format is being forced to innovate too. arXiv:2605.19999 proposed an extreme scheme: publish a benchmark without releasing the plaintext questions — publish each question's KV cache, penultimate-layer hidden states, and the plaintext answers instead. Running inference-based evaluation with these states works perfectly; anyone hoping to train on them is out of luck, because training needs complete token sequences, and without plaintext there is nothing to memorize. On LLaMA-2 7B, a 100k-token KV cache is about 50GB, which PyramidKV compresses to about 350MB — perfectly distributable. Leak prevention has moved up front, from "detect after release" to "design the release format."

Figure: A day of verification work (fingers hovering in mid-air between the "I swear it's fine" button and the "hand over provenance and traces" button)

05🧾 For buyers: what gets verified shifts from question text to physical evidence

Once the questions are invisible, what does a buyer inspect? The object of verification shifts from "question text" to "evidence of the production process." Put plainly, you check this batch of data's household register and health report. Five pieces of evidence: provenance (which upstream repository it came from, which commit it matches, whether the hash manifest checks out); real-run traces (trace logs from frontier models actually running the tasks — don't just take "the tasks are hard" on faith); elimination evidence (how many bad questions were filtered, how they were screened, and where the rejected ones went); the anti-leak process (whether the questions ever entered a public repository — this decides how long the data stays alive after purchase); and answer decidability (programmatic assertions in a sealed sandbox, formal verification, or maintainer-grade rubrics).

Figure: Three pieces of evidence on the inspection bench (a provenance bag labeled PROVENANCE, a film reel of real-run traces, and a chip bag stamped "eliminated," each verified one by one under the inspector's magnifying glass)

This is why we at DataMining treat these five categories of evidence as standard deliverables. Take our DeepSWE-class delivery package: alongside the schema 1.3 README and the 5 task bodies, it ships with validation.json containing 33 real run records from GPT-5.6 Sol; PROVENANCE.md documents the upstream commit, the official grader hashes, and the input delivery hashes; plus a SHA256SUMS checklist — flip a single byte and it fails on the spot. The scientific-computing task package includes selection.json: complete evidence for screening 3 questions from 8 candidates, with every rejected question archived; a screen recording of the trace viewer shows the model running 71 model turns and 70 terminal calls, about 260k output tokens, and still failing to finish after all 5 attempts exhausted their budgets. Whether the tasks are hard — the run records speak for themselves.

Benchmarked with GPT-5.6-sol on our Deep SWE-class task data: ACC lands about 15 percentage points below the same-type data from comparable data vendors, with interaction traces about 20 turns longer. Our delivery lines are aligned end to end with ALE-Bench, Frontier-Eng, HLE, Frontier SWE, Deep SWE, Program Bench, and APEX Agents, with sealed sandboxes, programmatic assertions, and expert rubrics throughout; the HLE-class expert-authoring line uses Z3/SymPy for formal verification, with stable capacity of 200 questions per week on a 7-business-day cycle.

06🎯 Summary

String the whole technical chain together: question banks publicly hosted → crawlers pull them into training corpora → cloze guessing exposes memorization (52%/57%) → clean mirrors squeeze out the inflation (up to 13 points) → 13-gram filtering bypassed by rewrites → evaluators rotate questions dynamically or lock them private → privatization cures leakage but not "exam-hall recognition" → grading shifts to maintainer-merge → the release format shifts to KV-cache. In an era when the exam papers are locked away, the only thing left to verify is real-run traces inside a sandbox.

"Public leaderboards are for the spectacle, private sets are for the substance, and verification accepts nothing but the evidence chain."

Similar work we have delivered

Case	What we did	Scale / turnaround
Deep SWE-class coding tasks	Real repositories, real verifiers, and reference solutions, delivered with PROVENANCE and real-run records	200 questions/week / 7 business days
HLE math-physics-chemistry set	Multimodal expert questions authored fresh by a competition-coaching team, math verified formally with Z3/SymPy	1,000–2,000 questions per subject / 3 business days
Scientific computing task package	Full screening evidence for selecting 3 from 8, with real-run trace viewer recordings	Per batch / 7 business days
Industry expert task packages	Authored by practicing experts in healthcare, law, and finance, with distractor files and rubrics, passing five-dimension consistency QA	Per project / per project agreement

DataMining (datamining.cloud) was founded by a PhD team from the Institute of Software, School of Computer Science, Peking University, and provides original task data for LLM and agent R&D across coding, complex reasoning, multimodal, and agent tasks. To build a private evaluation set, add high-difficulty original tasks, or request sample data and delivery evidence checklists, contact the team: contact@datamining.cloud

07References

1. Artificial Analysis Intelligence Index v5
2. Gemini 4 Carbon takes No. 1 on v5
3. TS-Guessing (arXiv:2311.09783)
4. MMLU-CF (arXiv:2412.15194)
5. Contamination-resistant benchmark design (arXiv:2605.19999)
6. Detector miss rate (arXiv:2601.19334)
7. Evaluation Awareness (arXiv:2505.23836)
8. MathArena (arXiv:2505.23281)
9. LMSYS deduplication blog post
10. OpenAI stops evaluating SWE-bench Verified
11. Terminal-Bench 4.0

💬 When was the last time your evaluation set got a full rotation? Tell us in the comments. If you found this useful, like and follow, and see you in the next one.