The Second Half of the AI Data Business: Buying Process, Rewriting the Exam
From buying answers to buying trajectories, from public leaderboards to private blind sets, and the embodied-data problem nobody has solved yet.
The first half of the AI data business was about volume: scrape, clean, label, ship. The second half is about evidence. Buyers no longer ask how many answers you have; they ask how the answer was produced, whether the exam can be gamed, and what happens when the task leaves the screen and enters the physical world.
01Buying the process, not the answer
A final answer is cheap to imitate and hard to verify. A trajectory is the opposite: it carries tool calls, intermediate states, failed branches and the point where the model changed its mind. Post-training needs that signal. Reinforcement learning on verifiable rewards works because environments can replay a run and decide pass or fail; a static answer cannot be replayed at all.
This is why delivery units have shifted from "question + reference answer" to task packages with an instruction, a runnable environment, a deterministic verifier, private tests and a provenance manifest. What the buyer acquires is a repeatable judgement procedure, not a text.
02Once the exam is public, everyone games it
Public benchmarks saturate. Contamination is not an edge case: once a test set is reachable, it enters pretraining corpora, prompt libraries and vendor demos. The remaining defences are procedural — private blind sets, holdout rotations, version discipline, contamination screening, and comparing pass@k under a fixed budget rather than quoting a single headline score.
Rubrics change the shape of the exam as well. A single overall score invites surface optimisation: longer answers, tidier formatting, more confident tone. A checklist of atomic criteria with evidence anchors and hard stops gives the judge something to verify instead of something to like. But rubrics are not automatically better — finer granularity can reduce judge agreement when aggregation is crude, so consistency must be measured, not assumed.
The deeper problem is reward hacking. If a verifier has a hole, the model will find it. Hardening an exam is an adversarial loop: red-team the verifier, patch the hole, confirm that legitimate solutions still pass, and repeat. A benchmark without that loop is a marketing asset, not a measurement instrument.
03The unsolved problem: embodied data
Text tasks can be replayed in a container. Physical tasks cannot. Collecting embodied data means real devices, real spaces, real variability and real safety constraints; every retake costs time and hardware, and the "same" demonstration is never quite the same trajectory. Teleoperation logs, grounding annotations and DAgger-style corrections each answer a different question, and no public benchmark currently settles how difficulty should be calibrated across them.
For now the honest position is to keep provenance: which scene, which operator, which sensor stream, which correction, which failure. Without that chain, a robotics dataset is a pile of clips rather than an evaluation instrument.
04What makes a vendor last
Repeat orders, not launch announcements. The vendors that survive the second half are the ones that can show acceptance records: contradiction QA before delivery, verifier hardening after hacking attempts, stable weekly throughput of genuinely new tasks, and the humility to publish limitations alongside results. Claims about expert headcount or token volume are unverifiable self-reports; a reproducible verifier and an auditable failure log are not.
About DataMining Founded by PhD researchers from the Institute of Software, School of Computer Science, Peking University, DataMining builds original task data and executable evaluation environments for frontier models and agents, with reproducible rubric specifications and bootable Harbor Task verifiers. Benchmarks named here are public third-party benchmarks; we align capability requirements without copying tasks. Visit datamining.cloud or write to contact@datamining.cloud.