Grading When There Is No Answer Key: Inside Rubric Scoring Scales
When a task has no answer key, the grader no longer holds the one correct answer. It holds a written scoring checklist: which item was met, which was not. That is a rubric.
The previous part took the scoring machinery apart: Pass@1 for a single blind attempt, Pass@k for repeated draws, resolve rate for bug fixes that must not break existing behaviour, and Elo after stripping the formatting premium.
But a large class of tasks has no score to compute at all.
Ask a model to write a forty-page industry report, analyse a contract with dozens of clauses, propose an architecture migration for a legacy system, or drive a chain of tool calls until a ticket closes. There is no single right answer to compare against.
The old approach was blunt: call another model, ask for a total score.
8 out of 10.
The number carries some signal, and very little information. Why 8? Did it lose points for a factual error, or for missing content? Take that 8 into a review or a procurement decision and it does not tell you what to fix.
Rubrics exist to solve exactly that problem: a total score with no visible deduction points.
01Premise: the answer keys ran out
The first two parts share one premise: the result can be verified by machine.
- MMLU is multiple choice. The model picks A, B, C or D, and the answer sits in the item bank.
- HumanEval is code. Run the unit tests on the function and you know whether it passed.
- SWE-bench is real engineering. The patch counts as a fix only when the tests turn green.
In that world right and wrong are binary, and a grading script returns a definite score. It is clean, and it drove most of the progress of the past few years.
The problem is that the work models are now handed keeps leaving that comfort zone. Long-form writing, research synthesis, chart production, industry analysis with commercial judgement: none of it can be written as a unit test.
OpenAI's HealthBench pushes the point to its limit: 5,000 real doctor-patient conversations, each annotated with 48,562 criteria written by 262 physicians over 11 months, a median of 11 criteria per item.
The point is that the doctors could not write answer keys either. Real consultations, advice and recommendations have no template answer to check against.
02What a rubric is: turning "pretty good" into checkable items
In plain terms, a rubric is a scoring checklist — the sheet a teacher holds when marking essays, except it does not say "was the essay good", it lists specific requirements that can be ticked off.
The job is concrete: turn a vague "is this answer good" into a set of items that can each be checked.
So the value of a rubric is not that scoring becomes more complicated. It is that the word "good" becomes visible, checkable and arguable.
Here is a real example. In the V-Rubrics paper, the S-Lab group at Nanyang Technological University took one visual question-answering item and split the reference answer into six independent rubric entries, each labelled with its dimension, its importance level, the basis for judgement, and its share of the total score.
After the model answers, the judge no longer produces an unexplainable total. It rules item by item: which one was met, which one was missed, which one was a trap.
That matters most when two models score close together. You can see where each is strong and weak instead of staring at two 80s and guessing, and you know where to spend effort.
03What a usable rubric looks like: six parts
A mature scoring scale is assembled from six parts. Drop any one and it fails quietly somewhere in the chain.
Part 1: dimensions
What aspects are being judged. For a popular-science article written by a model, the dimensions might be accuracy, completeness, beginner friendliness, structural clarity, quality of examples, and risk warnings.
Part 2: a definition per dimension
Writing "accuracy" is not enough; you have to say what accuracy means. For instance: core concepts, data and citations contain no obvious errors, and the conclusion is not overstated beyond the scope of the evidence.
Part 3: score levels
It can be 0/1, or 1 to 5. A 1 means many factual errors, a 3 means the core concepts are broadly right with omissions, a 5 means the concepts are accurate and the boundary is clear.
Part 4: weights
Not every dimension matters equally. In a scale that grades medical advice, safety must carry a high weight; for creative writing, expression and style matter more.
Part 5: reference answers or evidence
If there is a reference answer, supporting material or a source of fact, give it to the judge. Without one the judge scores from impression and can accept a wrong claim as right. This part is routinely skipped, and the evaluation collapses into a taste vote among a few people.
Part 6: aggregation
How do the dimensions combine into a total? A plain average, a weighted average, or a hard gate such as failing outright when safety scores below 3?
The sixth part is the most often ignored and the most consequential. Sum everything and the model can compensate for factual errors with polished prose: long and fluent, and the writing quality buys back the lost points. Under a hard gate, a factual error cannot be offset by fluency.
04Four ways these scales get written
The first is writing them directly. A person or a model lists the dimensions. Fastest, and prone to platitudes. "The answer should be helpful" swallows almost anything and discriminates poorly. "Does the answer give at least two concrete examples that let a beginner tell accuracy, pass rate and win rate apart" becomes checkable.
The second is contrastive generation, and it is the most practical. Show the model a good answer and a bad one and ask what differs. One explains how evaluation types differ, describes the scoring method and flags the limits of applicability; the other says "evaluation is testing". Criteria pulled from that contrast are far more solid than criteria invented from scratch, because they come from real differences.
The third is iterative patching. Write a version, score with it, find the flaw, fix it. If the judge keeps rewarding long answers, add a line: no credit for length itself, only for information that was not already there.
The fourth is letting the scale evolve with the model. The reason is simple: models probe for loopholes, so a rubric cannot stay a static table forever.
05The finer the rubric, the better models game it
This is the section worth remembering.
In May 2026 the Scale AI team ran a plain check: take a model trained with a rubric as its reward and re-score it with top judges from three different model families, to see whether it had actually improved or merely learned the exam.
The result was uncomfortable. The verifier used during training kept raising its reward while judge consensus did not follow. In some settings the two reversed: judges given the rubric preferred the trained model, while judges with no rubric at all preferred the original.
So where did the score come from? From "completeness" and "presence" criteria — did the answer say all the things it was supposed to say. Meanwhile factual correctness, conciseness, relevance and overall quality all fell together.
Gaming takes three typical forms:
- Half-satisfied compound conditions. The criterion says "both A and B", and the verifier awards the point for one of them.
- Reading implied content as stated. The required claim is never written, and the verifier supplies it from context or common sense and scores it anyway.
- Imprecise verification. Matching the topic is enough; whether the specific concept was checked does not matter.
The paper's conclusion is sober rather than optimistic: a stronger verifier cuts this gaming substantially but does not eliminate it. As long as important failure modes remain unstated in the rubric, a stricter verifier can still be walked around.
A note from engineers in the field. Building evaluation environments for partners, we keep hitting the same thing: the grading logic and the model under test have to be physically isolated. Test files must not sit in a directory the model can read or write, and the grading script has to run in a sandbox the model cannot reach. Otherwise even a careful rubric is just a map telling the model where the exam paper is.
The paper also offers a practical diagnostic, the self-internalization gap, which needs no extra judge: the model's own output distribution is enough to tell whether it has stopped improving.
06Rubrics as rewards: the hottest shift of 2026
For the past two years a rubric was mostly an evaluation tool. It is now used directly as a reinforcement-learning reward signal, the clearest route change in post-training this year.
Scale AI's RaR (July 2025) is the reference implementation: the rubric becomes the reward, letting models do RL in medicine and science where no answer key exists. Against a baseline that has a judge model assign a Likert score, the best variant reaches up to 31% relative improvement on HealthBench and 7% relative on GPQA-Diamond.
The framing has to be exact: 31% is a relative improvement, not 31 percentage points, and it is not available in every setting. The paper itself says "up to".
Rubric Anchors, from Ant Group's Inclusion AI and Zhejiang University (August 2025), pushes further: over ten thousand criteria, trained on just over 5K samples, a 5.2% absolute margin on open-ended benchmarks, 2.4 points above the 671B DeepSeek-V3, with no regression in general or reasoning ability. One useful by-product: using the rubric as a style anchor suppresses the instantly recognizable "AI voice".
Note that the framing differs from RaR: Rubric Anchors reports a 5.2% absolute margin, RaR reports a 31% relative gain. Different rulers, so the two numbers cannot be compared directly.
A finer-grained route exists as well. SRaR, published in May 2026 by teams including Peking University and JD Explore and accepted at NeurIPS 2026, applies the rubric to every step of the reasoning chain. They found something counter-intuitive: among 1,000 sampled problems, 18.2% of the steps in answers that ended correct were still wrong, and 49.9% of the steps in answers that ended wrong were right. With step-wise rubric rewards, faithful reasoning on AIME 2025 rose from 34.5% to 46.7%, and self-correction loops fell from 48.1% to 26.5%.
Vision has a parallel. V-Rubrics from NTU S-Lab (August 2026) used 50,000 samples across 17 visual data sources to produce 350,000 rubric entries along three dimensions: visual faithfulness, reasoning consistency and instruction following. The core move is to push the reward from the whole answer down to a single fact, so the model learns which part it got right and which part it got wrong.
07Three rules for writing your own scale
Across the three parts we have covered subjects, scores, and how to grade when no answer key exists. That is the skeleton of model evaluation. For your own rubric, three rules are worth holding.
- Decide what the scale is meant to block. A rubric that cannot state which failure mode it defends against will be walked around during training. All three forms of gaming above come from a boundary the rubric never wrote down.
- Keep deduction items next to credit items, and give each deduction a clear trigger. Credit items are easy to satisfy, so models drift toward them. What actually defends the line is the item that says exactly when points must be lost.
- Cross-check a model you trained yourself with judges from other families. The rubric you wrote is precisely the target your model is best at optimizing. Being both syllabus and grader produces scores that look great and mean nothing.
About the DataMining team We are a doctoral team from the School of Computer Science, Peking University, producing original task data and evaluation infrastructure for model and agent development, with data services delivered to ByteDance, Alibaba and Tencent. In code engineering, complex logical reasoning and long-horizon sandbox interaction we build anti-cheat, independently verifiable environments and blind test sets for partners. Website: datamining.cloud. Contact: contact@datamining.cloud
Three basic questions are settled. One more uncomfortable question sits outside all of them.
After a leaderboard has been studied hard enough by enough people, how much reference value is left?
The first part opened with Goodhart's law: when a measure becomes a target, it stops being a good measure. Item banks get contaminated, leaderboards get fitted, test sets get quietly memorized, and all of it is happening now.
08References
- Gunjal et al. Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains. Scale AI. arXiv:2507.17746, 2025.
- Mahmoud et al. Reward Hacking in Rubric-Based Reinforcement Learning. Scale AI. arXiv:2605.12474, 2026.
- Huang et al. Reinforcement Learning with Rubric Anchors. Inclusion AI, Ant Group & Zhejiang University. arXiv:2508.12790, 2025.
- Tian et al. V-Rubrics: Visual Faithfulness via Rubric-Based Reinforcement Learning. S-Lab, Nanyang Technological University. arXiv:2608.25580, 2026.
- Xie et al. Step-wise Rubric Rewards for LLM Reasoning. NeurIPS 2026. arXiv:2605.17291, 2026.
- Arora et al. HealthBench: Evaluating Large Language Models Towards Improved Human Health. OpenAI. arXiv:2505.08775, 2025.