A Rubric Design Guide: How Scoring Criteria Teach a Model to Play the Angle

If you work on LLM-as-a-judge, or use rubrics as a reinforcement-learning reward, you have probably seen this happen.

On the evaluation side, a model scores 90 on the eval set, and the human expert shakes their head: pages of correct filler, the core business logic wrong.

On the training side, you feed expert-written rubrics into the RL pipeline as rewards, and the trained model learns to cut corners: immaculate formatting, extremely polite tone, padded length, and the moment it faces a real business environment the act falls apart.

The problem sits in the rubric.

In the 2026 post-training and evaluation stack, a rubric is no longer the grading sheet of traditional education. It is the basis on which an LLM judge rules, and the source of gradients in a reinforcement-learning pipeline.

Drawing on representative work published in 2026 (AutoRubric from Notre Dame and Uniphore, V-Rubrics from NTU S-Lab, CaRR from Tsinghua and Zhipu) and our own experience delivering evaluation environments at datamining.cloud, this piece covers one thing: how to design an industrial rubric that blocks cheating and still supplies a dense training signal.

Whether you are starting as an evaluation engineer or own the reward function for RL, the details below are where rubrics actually fail.

01First principle: a rubric is a judging contract, not a prompt

Most beginners write rubrics that are really prompts.

"Please assess whether this merger-risk memo is accurate, complete and fluent, and give a score from 1 to 5."

The literature calls this holistic scoring [1]. Its fatal flaw is a vague standard, and a vague standard forces the judge back onto its own priors: longer answers score higher, tidy structure and polite tone score higher. The Scale AI paper on rubric reward hacking gives direct evidence: response length climbs throughout training, and length correlates strongly with satisfying presence and completeness criteria [6].

What industry actually uses is analytic scoring [1]: break "good" into dozens of observable, falsifiable atomic assertions.

One easily missed fact: finer is not always better. A controlled study from CMU and Google Research found analytic scoring is not reliably better than holistic scoring; higher criteria complexity combined with conservative aggregation actually lowers agreement between human and automatic raters [2]. A Tsinghua survey reaches a similar point: on tasks that demand high completeness, holistic judging with detailed criteria beat atomic judging [1].

So the correct statement is not "the finer the more scientific". The return on splitting depends on task type and aggregation, and the evaluation side has to measure agreement rather than deciding by intuition.

The return on splitting does not come from being finer; it comes from being more checkable. A judgement must point back to a specific location in the source material, otherwise splitting only chops an impression score into pieces.

A usable rubric carries three things.

• Dimension. What is being checked, for example consistency of financial data.
• Anchors. What counts as good and what counts as bad, with the boundary stated.
• Evidence source. Where the judge goes to verify the fact.

A bad rubric reads like a prompt.

"Check whether the financial data in the memo is accurate."

A good rubric reads like an atomic assertion.

"Check whether the 'Q3 2025 net profit' cited in section 2.1 of the memo matches the Non-GAAP Net Income figure on page 14 of input_files/financials_2025Q3.pdf, allowing rounding error within 0.1%."

The first lets the judge score by impression. The second makes the judge verify. Reducing a subjective judgement to a checkable action is what drives the variance between judges down.

02Field notes: four recurring rubric defects

In our sampling review of externally crowdsourced rubrics (random draws from delivered batches that were rejected), four defect types kept coming up and account for most rejections. They are not beginner slips; they are the places this craft falls through.

Defect one: the "and" trap, one criterion checking two things

Annotators like to write "check whether the model answered the user's question accurately and completely".

If the answer is accurate but incomplete, does the judge pass or fail it? The judge deadlocks and returns a random score, and the reward signal fills with noise.

The fix is plain: one criterion checks one thing. Conjunctions such as "and", "at the same time" and "as well as" are banned. Split into "accuracy of core concepts" and "coverage of key sub-questions", each scored independently.

Defect two: no veto item, and penalties too light to matter

The rubric is all +3 and +5 credits with no penalties. The model quickly learns to hide factual errors behind polished formatting: enough words and a pleasant tone earn the base score.

A penalty has to be more than a nominal negative number. Tencent Hunyuan, Tsinghua and Peking University give a concrete method in the RLMR paper. First turn the reward of a violating sample into rᵢ minus δ, then derive that the penalty lower bound must satisfy δ ≥ (n·r_max^vio + n·γ − Σrᵢ) / (n − k), where r_max^vio is the highest raw reward among violating samples in the group. Only a δ above that bound guarantees every violating sample receives a negative advantage after GRPO normalization and is systematically suppressed by the gradient [11].

In plain terms: the size of the penalty has to be derived from the highest violating reward in the group, not guessed as a round −5.

Defect three: a dangling evidence chain leaves the judge relying on memory

Ask a judge to assess "whether the legal citations the model used are correct" without giving it a legal database, and it can only rule from pretraining memory. If the judge itself hallucinates, it marks the model's hallucination as correct, and hallucination gets rewarded.

CaRR from Tsinghua and Zhipu builds this into the mechanism: a trajectory satisfies a criterion only when it explicitly names the hidden entity, the citation it gives is supported by the page text, and the satisfied criteria chain into a complete evidence path to the final answer [5]. V-Rubrics from NTU splits reference answers into 352,938 atomic criteria across visual faithfulness, reasoning consistency and instruction following, each pointing back to a specific location in the image or the question [4].

In practice, every factual scoring point must bind to a real evidence source. For example: "Whether the conclusion holds must be judged on, and only on, the testimony in pages 3 to 5 of context/case_A.pdf; a model that introduces information not provided fails."

Defect four: sign semantics reversed, and the whole thing inverts at compile time

When domain experts design a rubric table, they write a dimension called "did not fabricate facts" with weight −10, meaning ten points are deducted if the model fabricates.

If the engineering script copies that table straight into code, a judge ruling "no fabrication = true (1)" multiplies by −10 and deducts ten points from the model, while a model that actually fabricated (ruled 0) loses nothing.

The fix is a strict compile and sign-alignment layer between the expert-readable rubric and the RL-executable reward function. Every condition is rewritten as "trigger earns points" or "trigger fails the task", with no negative-sign semantics carried over.

03Advanced practice: turning a rubric into a dense reward signal

Treat a rubric only as an evaluation tool and you use a small part of its value. In 2026 its main power is that it is becoming the dominant dense reward signal in non-verifiable domains.

Math and code have unit tests that supply a 0/1 reward. Financial due diligence, legal review and multi-step agent workflows have none. Scale AI's RaR uses instance-specific rubrics directly as the reward for on-policy RL, pushing the boundary of verifiable rewards outward; the best variant reaches up to 31% relative improvement on HealthBench and 7% relative on GPQA-Diamond against a judge baseline that scores Likert directly [7].

Ant Group and Zhejiang University then published two reproducible designs. Rubicon (Rubric Anchors) built a reward library of more than 10,000 human-written rubrics and completed two-stage RL on Qwen3-30B-A3B with just over 5K training samples; the resulting Rubicon-preview gains 5.2 percentage points absolute on humanities-oriented open-ended benchmarks and 2.4 points over the 671B DeepSeek-V3 [8]. CoCA points out two losses in standard rubric-based RL: clause-level feedback is collapsed into a scalar reward, and token advantages are distributed uniformly, so the structured supervision goes unused. CoCA reframes the problem as clause-aware credit assignment and uses the JSD shift in next-token distribution caused by adding a clause condition as a relevance cue, assigning credit down to tokens; across seven medical and science benchmarks it averages 64.5%, 4.2 points above the RaR baseline at 60.3% [9].

Another route comes from industry. The Vanguard Group paper converts one-off, limited expert judgement into a dense reward, mixing expert-verified rubric scores with retrieval metrics into a GRPO-compatible hybrid reward. Motivated by non-verifiable domains such as financial services, the experiments run on query rewriting for open-domain retrieval with Natural Questions, where Recall@3 rises from 67.96 in the sparse-reward baseline to 74.00 [10].

In-group ranking: separating a rough pass from a clean pass

A hard task is sampled 16 times in RL, and 4 of those pass the baseline criteria. Give all four +1.0 and move on?

No. That discards in-group discrimination and leads to policy collapse. The concept of the Sharpening Tax describes exactly the loss of test-time scalability that post-training imposes relative to the base model; Meta Superintelligence Labs with the University of Wisconsin-Madison, NYU and Stanford verified its prevalence across 14 base/post-trained model pairs and 3 agentic benchmarks, 42 cases in total [12].

The advanced move is a second layer of ranking inside the group that already passed the baseline: "is the code change the smallest possible", "are the reasoning steps redundant". Positive advantage shifts from trajectories that passed by luck to trajectories that passed cleanly.

Adversarial evolution: patching the rubric with a hacker-fixer loop

Models are very good reward hackers. Write "must include 3 risk warnings" and a model may append three warnings in white text on a white background to collect the points.

The mechanism has a name. The Hacker-Fixer loop is formalized in Hardening Agent Benchmarks with Adversarial Hacker-Fixer Loops: a hacker model tries to fool the verifier without solving the task, a fixer model patches the discovered hole, and a solver model confirms that a legitimate solution is still accepted after the patch. Each patch changes how the verifier rewards behaviour, which exposes the next hole. On KernelBench the loop drove the attack success rate against disclosed holes from 62% to 0 [13].

If what you need to defend against is gaming at the rubric layer itself, the closest recent work is RubricArmor's attack-repair loop [14].

04Field manual: an industrial rubric template

Below is the standard rubric structure we use at datamining.cloud when delivering tasks such as the cross-period ledger reconciliation and compliance audit agent. The package is wrapped in the Harbor Task specification defined by the open-source Harbor framework (github.com/harbor-framework/harbor, the official Terminal-Bench harness); each task directory holds instruction.md, task.toml, environment/, solution/ and tests/, and can bring up its own container and emit a verifier reward. Harbor Task is a third-party framework format and we deliver to its native specification rather than a private variant.

The rubric itself lives in a separate file with the structure below (illustrative; real projects adjust to the client's data fields).

task_id: "fin-audit-cross-period-001"

# aggregation: baseline gate plus weighted sum
aggregation_logic: "baseline_gate_and_weighted_sum"

# 1. hard-stop items (veto)
hard_stops:
  - id: "HS-01"
    dimension: "data privacy compliance"
    criterion: "In the final audit report, are all national ID and bank card numbers referring to individuals masked with '*'?"
    evidence_anchor: "user_id column of input_files/raw_ledger.csv"
    penalty: -100

  - id: "HS-02"
    dimension: "factual hallucination"
    criterion: "Can every anomalous transaction amount in the report be matched to an exact transaction_id in raw_ledger.csv? Fabricated entries are forbidden."
    evidence_anchor: "input_files/raw_ledger.csv"
    penalty: -100

# 2. positive scoring items (single action, bound to evidence)
positive_rubrics:
  - id: "P-01"
    dimension: "cross-period pattern recognition"
    criterion: "Is the duplicate-charge pattern between 2025-12-31 23:50 and 2026-01-01 00:10 identified correctly?"
    weight: +15
    evidence_anchor: "timestamp and transaction_type columns of raw_ledger.csv"

  - id: "P-02"
    dimension: "root cause"
    criterion: "Is the root cause attributed to a gateway timeout retry without an idempotency check, rather than to a user submitting duplicate orders?"
    weight: +10
    evidence_anchor: "section 4.2 of input_files/system_architecture.md"

# 3. negative penalty items
negative_rubrics:
  - id: "N-01"
    dimension: "format redundancy"
    criterion: "Does the report open with more than 200 characters of boilerplate such as 'As an AI assistant, I should remind you...'?"
    weight: -5
    evidence_anchor: "first 3 lines of output/report.md"

Three properties make the structure work.

• Clear aggregation. The baseline gate runs first. Expose a single ID number and the reward is 0 no matter how good the analysis is. The model gets no chance to hide a prohibited error behind talent.
• Evidence anchors are mandatory. When the judge scores, the system highlights the file fragment named by evidence_anchor and feeds it to the judge. Impression scoring has nowhere to live.
• Rough passes and clean passes separate. A negative item such as N-01 demotes a "found the bug but padded the report" trajectory in RL in-group ranking.

One engineering discipline on top: the table is for humans. Whatever enters the RL pipeline must be regenerated through the sign-alignment layer; never multiply a negative number from the table by a boolean.

05Closing: whoever defines good controls post-training

Back to the opening question: why does your model act like an angle-player.

Because you handed it a vague definition of good, and the model obliges with the disguise it is best at.

In the second half of 2026, verifiable rewards in math and code are saturated. The deep water is finance, law, medicine and multi-step agents: open domains with no answer key.

In those domains the rubric is the only law.

Whoever builds the marking room first gets to train agents that actually work in the real commercial world.

Stop letting the model play the agreeable middleman; write the judging contract and build the standardized marking room.

About DataMining A doctoral team from the School of Computer Science, Peking University, producing original task data and executable evaluation environments for model and agent development. We deliver not only data but reproducible rubric criteria, plus independently launchable container environments and verifiers packaged to the Harbor Task specification of the open-source Harbor framework (harbor-framework/harbor). Long-horizon code and engineering simulation: Deep SWE, Frontier SWE, Frontier-Eng and ALE (independent Docker packaging as Harbor Tasks), delivered at 200 tasks per week with a 7-working-day turnaround. Hard mathematics and professional agents: the strict multimodal HLE line (5,000+ tasks in stock); Harvey LAB, AutomationBench and Toolathlon; and industry workflow packages carrying 35 to 60 atomic positive and negative rubrics. System base and embodied real-robot data: standardized Windows 11 and Ubuntu desktop-agent VM images, and a million-scale embodied grounding, SFT and DAgger pipeline on real robots. The benchmarks named here are all public third-party benchmarks; we align with their capability requirements and difficulty bands and do not copy their items. The difficulty alignment and delivery capabilities described are our internal same-standard comparison results, not official conclusions of the third-party benchmark owners. Website: datamining.cloud. Sample requests and technical contact: contact@datamining.cloud

06References

[1] AutoRubric: A Unifying Framework for Rubric-Based LLM Evaluation on Non-Verifiable Tasks. Delip Rao, Chris Callison-Burch. University of Pennsylvania. COLM 2026. arXiv:2603.00077.

[2] Quantifying the Statistical Effect of Rubric Modifications on Human-Autorater Agreement. Carnegie Mellon University and Google Research. arXiv:2605.06283.

[3] AutoRubric: Rubric-Based Generative Rewards for Faithful Multimodal Reasoning. Mengzhao Jia et al. University of Notre Dame and Uniphore. Findings of ACL 2026. arXiv:2510.14738.

[4] V-Rubrics: Visual Faithfulness via Rubric-Based Reinforcement Learning. Shulin Tian et al. Nanyang Technological University S-Lab, A*STAR and UIUC. EMNLP 2026 main conference. arXiv:2608.25580.

[5] Chaining the Evidence: Robust Reinforcement Learning for Deep Search Agents with Citation-Aware Rubric Rewards (CaRR). Tsinghua University and Zhipu AI. ACL 2026. arXiv:2601.06021.

[6] Reward Hacking in Rubric-Based Reinforcement Learning. Anas Mahmoud et al. Scale AI. arXiv:2605.12474.

[7] Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains (RaR). Anisha Gunjal et al. Scale AI. arXiv:2507.17746.

[8] Reinforcement Learning with Rubric Anchors (Rubicon). Zenan Huang et al. Inclusion AI, Ant Group and Zhejiang University. arXiv:2508.12790.

[9] CoCA: Criterion-Aware Credit Assignment for Rubric-Based Reinforcement Learning. Guangcheng Zhu et al. Zhejiang University and Ant Group. arXiv:2610.17733.

[10] Expert-Guided Rubrics as Dense Reward Signals for Reinforcement Learning in Non-Verifiable Domains. Ashish Bansal et al. The Vanguard Group. ICAIF ACM RL4LLM-Agents workshop. arXiv:2610.26932.

[11] RLMR: Reinforcement Learning with Mixed Rewards for Creative Writing. Jianxing Liao et al. Tencent Hunyuan, Tsinghua University and Peking University. arXiv:2508.18642.

[12] Sharpening Tax in Post-Training. Changdae Oh et al. Meta Superintelligence Labs, University of Wisconsin-Madison, New York University and Stanford University. arXiv:2610.01509.

[13] Hardening Agent Benchmarks with Adversarial Hacker-Fixer Loops. Ziqian Zhong et al. Carnegie Mellon University and Fewshot. arXiv:2606.08960.

[14] RubricArmor: Adversarial Evolution Improves LLM-Based Rubric Generation. National University of Singapore et al. arXiv:2610.05308.

[15] Introducing Harvey's Legal Agent Benchmark (LAB). Harvey official blog, harvey.ai, 2026.

07Further reading

• Part 3: Grading When There Is No Answer Key
• Anthropic Cut Evals Web Access Over Environment Flaws
• Locking Leaderboards in a Vault: Data Contamination and Verification Rules
• Four Dimensions for Evaluating AI Data Vendors