35 Atomic Rubrics per Task: Agent Data Procurement Now Buys Grading Rules
One task needs at least 35 atomic checks. Inside the quantity floors, judge agreement thresholds and positive/negative weighting that now define Agent data acceptance.
In October 2026 we revised the task specification we send to domain experts. One thing changed: we replaced "looks hard" with "can be judged as failing by a machine". The acceptance lines are not industry rumours; they are clauses written into delivery contracts, acceptance sheets and QA reports.
01Buyers no longer grade answers. They gate models.
The shift across Q4 2026 is measurable. Harvey LAB ships 1,250 tasks across 24 practice areas with more than 75,000 expert-written criteria, roughly 60 atomic checks per task, all of which must pass. AutomationBench scores with paired positive and negative assertions across 600 public and 600+ private tasks, covering about 500 API endpoints in 47 apps; the strongest frontier model stays below 10% end-to-end. Toolathlon puts 604 tools from 32 apps into 108 long-horizon tasks, where the best reported result is 38.6%.
The common denominator: grading moved from "an expert reads it and gives a score" to "a machine reads it and reproduces the same verdict".
What the buyer used to purchase was word count. What they purchase now is a rule that can fail a frontier model.
02Why one task needs at least 35 separate checks
A professional Agent task is not a text artefact; it is a decidable artefact. Four requirements make it work.
Quantity floor. No fewer than 35 independent scoring points per task. Below that, a model can satisfy the checklist without doing the job.
Atomicity and verifiability. One criterion checks exactly one thing. "Accurate and complete" is forbidden; it becomes "core concept is correct" and "all required sub-questions are covered", each with its own evidence anchor.
Positive and negative weights. +10 / +7 / +3 for value delivered, −10 / −7 / −3 for business hazards, plus hard-stop clauses such as an unmasked ID number or a missing risk disclosure. A single failure can void the whole answer.
Provenance. Tasks are not invented by an expert at a keyboard. They come from real workflow deliverables, with decoys and hidden traps preserved, so the difficulty reflects the workplace rather than the author's imagination.
03The real nightmare is grading: the 0.70 and 0.85 lines
Writing criteria is the easy half. Calibrating the judge is harder. We require a candidate task to hold a score separation above 0.70 and a judge–expert agreement above 0.85 before it enters a delivery pack.
Two traps sit underneath those lines. First, the grader itself makes mistakes: a verifier can pass a wrong patch or reject a correct one, which is why every pack carries human re-adjudication samples and a misclassification rate. Second, double penalisation: the same defect must not be charged to two criteria, or the reward signal silently doubles a single error. We run a contradiction QA pass over prompt completeness, identifier consistency, reference-deliverable correctness and rubric polarity before anything ships.
A longer task is not a harder task. Difficulty comes from discriminative checkpoints, not from page count.
04Why loose expert networks cannot deliver this
Domain experts know their field, but they usually do not know how a model fails, and they rarely write program-level atomic criteria. Producing 35 checks that a script can evaluate requires model trial runs, score compression iterations and verifier hardening — a pipeline, not a freelancer. That is the actual moat in Agent task data.
The consequence for buyers is straightforward: ask for the rubric file, the evidence anchors, the judge agreement numbers and the failure records. If a vendor only shows you clean outputs, you are buying the numerator without the denominator.
About DataMining Founded by PhD researchers from the Institute of Software, School of Computer Science, Peking University, DataMining builds original task data and executable evaluation environments for frontier models and agents, with reproducible rubric specifications and bootable Harbor Task verifiers. Benchmarks named here are public third-party benchmarks; we align capability requirements without copying tasks. Visit datamining.cloud or write to contact@datamining.cloud.