Rubrics for Open-Ended Tasks: Evaluation and Post-Training in Practice
Atomic checks, evidence anchors, binary verdicts and hard stops: how industrial rubrics wire into LLM judges and dense RL rewards, with three deployment scenarios.
An industry research report, a medical diagnosis write-up, or a multi-step tool-use trajectory has no answer key. Exact match does not apply, unit tests do not exist, and a 1–5 holistic impression score rewards length, politeness and layout rather than judgement. Rubrics are the practical way out: an explicit set of checkable, independent criteria that turns "is this good?" into a list of verdicts.
01What a rubric actually is
A rubric decomposes a fuzzy quality judgement into atomic assertions about factuality, completeness, safety and reasoning reliability. If the benchmark is the exam paper and the metric is the final score, the rubric is the marking scheme held by the examiner. It decides not only how points are added up but what counts as a good answer in the first place.
That distinction matters because an LLM judge without a rubric falls back on its own priors. It favours longer, tidier, more agreeable outputs. A judge with a rubric performs verification instead: it checks a named artefact against a named source and returns a binary verdict.
02Four elements of an industrial rubric
Single-check atomicity. One criterion, one fact. Conjunctions such as "and" or "at the same time" are banned; they create deadlocks where a partly correct answer has no defined score.
Evidence anchor. Each criterion names the deliverable file, section or reasoning step it inspects. "Check whether the figures are right" is not executable. "Check whether the Q3 net profit in section 2.1 matches page 14 of the supplied filing, within 0.1% rounding" is.
Binary verdict plus weight. All-or-nothing satisfaction, then +10 / +5 for value and −10 or hard-stop for hazards. In medical and compliance work, one unmasked identifier or one missing risk disclosure can void the answer.
Hard stops and negative items. Positive-only rubrics are gamed by surface polish. Negative items must be heavy enough to survive group normalisation in GRPO-style training, not decorative −1 penalties.
03Three scenarios where rubrics earn their keep
Multimodal chart analysis. Criteria bind each numeric claim to a specific chart region, so a model cannot produce a fluent summary of numbers it never read.
Deep research and retrieval. Every scoring point must be tied to a real citation whose source page supports the claim. This is the mechanism behind citation-aware rubric rewards such as CaRR and the evolving rubrics in DR Tulu: an unsupported inference cannot earn full credit even when it reads coherently.
Legal and financial long documents. Checklists target omitted risk disclosures, unsupported assumptions and missing cross-period reconciliations — the failures that holistic judging systematically overlooks.
04Wiring rubrics into training and evaluation
In evaluation, rubrics give an auditable breakdown: factuality good, structure clear, evidence thin, safety boundary missing. In post-training, the itemised verdicts aggregate into a dense reward that replaces the 0/1 signal unavailable in non-verifiable domains. Work such as RaR, Rubicon, CoCA, V-Rubrics and expert-guided rubrics all follow this shape: decompose, judge per item, aggregate, update.
Two engineering disciplines are non-negotiable. Compile the human-readable rubric into an executable reward function with explicit sign alignment, so a negative weight never multiplies a boolean "condition satisfied" into a reward for the wrong behaviour. Then keep iterating: models find loopholes, so rubrics need contrastive generation from good/bad answer pairs, iterative patching after trial grading, and dynamic co-evolution as capabilities change.
05A short self-check for teams
Does every criterion check exactly one thing? Does it name its evidence source? Are hard stops heavy enough to dominate? Is there at least one negative item per business hazard? Can a judge's verdict be reproduced by another judge? If any answer is no, the score is still an impression.
About DataMining Founded by PhD researchers from the Institute of Software, School of Computer Science, Peking University, DataMining builds original task data and executable evaluation environments for frontier models and agents, with reproducible rubric specifications and bootable Harbor Task verifiers. Benchmarks named here are public third-party benchmarks; we align capability requirements without copying tasks. Visit datamining.cloud or write to contact@datamining.cloud.