No Answer Key: How Rubrics Became Dense Rewards for Post-Training
When tasks have no answer key, explicit rubrics replace holistic impression scores — first as marking schemes for LLM judges, then as dense reward signals for RL.
An industry research report, a complex medical diagnosis, or a multi-step agent run has no single correct answer. It is neither an MMLU multiple-choice item nor a HumanEval function with unit tests. So how should such work be graded — and how should a model be trained on it?
The default answers have been human preference alignment (RLHF) and LLM-as-a-Judge with a 1–5 holistic score. Both have a visible side effect: the model becomes a people-pleaser. Immaculate formatting, courteous tone, pages of correct-sounding filler — and no defensible judgement where the business actually lives. The concept both academia and industry have converged on is the rubric.
01What a rubric is, and why it matters now
A rubric decomposes "is this answer good?" into a set of explicit, independent, individually checkable criteria. A recent survey from Renmin University, The Rules of the Game: A Survey of Rubrics for Large Language Models, argues that as LLMs move into reasoning, decision-making and tool use in high-risk, long-horizon settings, simple correctness signals and unconstrained preference scores are no longer sufficient. Rubrics make implicit quality standards — factuality, completeness, safety, reasoning reliability — explicit.
If the benchmark is the exam paper and the metric the final score, the rubric is the examiner's marking scheme. It decides not just how points are computed but what a good answer is.
02The new consensus: rubrics as dense RL rewards
Reinforcement learning with verifiable rewards (RLVR) has been spectacular in maths and code because 1 for right, 0 for wrong is unambiguous. Most high-value real work — financial due diligence, legal compliance, medical analysis — cannot be asserted by a few lines of code. So teams are decomposing the expert reviewer's mental checklist into dozens of atomic rubric items that can each be scored, then aggregated into a dense reward.
- Vision and multi-step reasoning. V-Rubrics (NTU S-Lab) splits visual reasoning into visual faithfulness, reasoning consistency and instruction following, feeding itemised verdicts into a GRPO pipeline.
- Finance and deep research. Expert-guided rubrics turn intent understanding in financial retrieval into dense expert signals; Ai2's DR Tulu and Tsinghua/Zhipu's CaRR bind each scoring point to real citation evidence chains to suppress hallucination.
- Vertical deployment. Harvey's legal agent benchmark ships with tens of thousands of hand-written professional criteria, roughly 60 checks behind every document reviewed.
The survey's standard Rubric-based RL flow makes the change concrete: user input → policy samples multiple rollouts → an LLM judge scores r1…rK item by item → weighted sum produces rewards → policy update. Itemised signals replace a single binary verdict.
03What an industrial rubric requires
Popular explanations list dimensions, levels and weights. Production specifications are stricter.
Atomicity. One criterion checks one thing. No "and", no "simultaneously". "Accurate and complete" must become two separate checks.
Explicit deliverable anchors. Name the exact file or reasoning step the judge inspects.
Opposing rewards and penalties. +10 and +5 for value, but also heavy penalties for business hazards — a missed patient ID in a redaction list, or an omitted risk disclosure, can trigger −10 or an outright veto.
Mandatory evidence chains. Factual criteria require checkable citations. A coherent inference with no verifiable source cannot earn full credit.
The point is to shatter fuzzy human experience into fine-grained atomic assertions. Only then does an LLM judge stop wobbling and can the RL gradient point at a specific defect.
04How good rubrics get built
Asking a model to "generate a scoring standard" produces generic filler. Three routes work: contrastive generation, where the model compares a good and a bad answer to the same question and extracts the real differences; iterative refinement, where a first rubric is trial-graded and patched — add "length earns no points" if verbosity wins, add "citations must be verifiable" if references are invented; and dynamic co-evolution, because models game rules. When a model satisfies the surface wording without improving quality, the rubric must evolve with it, as DR Tulu does.
The survey's Figure 2 summarises these as direct generation, contrastive generation, iterative refinement and online co-evolution.
05From finding questions to defining standards
"Good" in the real world is not one-dimensional. An answer can be factually right but incomplete, fluent but weakly evidenced, useful but silent on risk. Rubrics make those differences discussable and traceable: factuality good, structure clear, evidence thin, safety boundary missing.
The competitive centre of post-training is shifting. It used to be who could scrape more maths and code problems with answer keys. Now it is who can translate the tacit expertise of their own domain into atomic scoring points a judge and an RL pipeline understand. That is not just an evaluation upgrade; it is how the industry becomes standardised enough to build agents that actually do work.
About DataMining Founded by PhD researchers from the Institute of Software, School of Computer Science, Peking University, DataMining builds original task data and executable evaluation environments for frontier models and agents, with reproducible rubric specifications and bootable Harbor Task verifiers. Benchmarks named here are public third-party benchmarks; we align capability requirements without copying tasks. Visit datamining.cloud or write to contact@datamining.cloud.