Public Leaderboards Are Billboards; Private Test Sets Are the Cash Register
What data vendors actually sell is mock exams and verifiable reward signals. Repeat orders come from reproducing a lab's internal failure cases, not a public score.
A year ago, the most questioned companies were the AI application startups rising too fast with valuations in the hundreds of millions. This year a different group looks almost restrained: data vendors founded less than a year ago have become multi-billion-dollar unicorns, while others sit in the tens of millions. Meta put more than $14B into 49% of Scale AI; Surge passed $1B in revenue; Mercor reached a $10B valuation. What keeps that money coming is not the public leaderboard. It is the willingness of frontier labs to keep paying for data and environments that produce a signal on test sets nobody else can see.
Strip away the grand narrative and this is a rough, fast-growing B2B trade. Vendors are workshops, not oligarchs. Scale, Surge and Mercor are the visible names, but the vendors that win second and third orders are usually the ones that can reproduce a lab's internal failure cases.
01What is actually being sold
The product, in plain terms, is mock exams and verifiable reward signals. They either find the model a tutor — collecting real expert problem-solving trajectories — or build the question bank: tasks, execution environments and grading rules packaged as RL environments, then sold to model labs for targeted training.
Pretraining consumes public, static internet text. In real professional work, models still fall short of a qualified practitioner, so they need domain experts and deliberate practice. That gap is the entire business.
02Benchmarks as a sales vehicle
Chinese vendors have developed a distinctive move: publish a benchmark. This is not academic evaluation; it is precision marketing aimed at model labs — "your model is failing our terminal-engineering exam; the lab next door topped the class because it trained on our question bank".
For two years, labs competed on compute and algorithms. Both now have diminishing marginal returns. Once the pipeline is built, the remaining bottleneck is raw material: data. That is why evaluation and data procurement have become the same conversation.
03Why the private set, not the public one
A public score is a billboard: it must be legible to investors, press and recruits. A private test set is the cash register: it decides whether the delivery is accepted and the contract renewed. A vendor that can show a failure case reproduced inside a lab's own environment, with a versioned task, a deterministic verifier and an auditable result, has something a leaderboard cannot substitute.
This also explains the industry's social structure. Researchers at frontier labs come from a small set of universities and labs; colleagues, alumni and former teammates move between labs and vendors. Founders of leading data vendors frequently come out of model labs, and after Meta's investment in Scale, some labs shifted noticeably toward Surge and Mercor. Distribution runs on trust and relationships as much as on technical specs.
The public contest is only the visible tip. A much larger amount of work happens in unwritten territory: model distillation, laundering of private data, and evaluation setups designed to capture resources rather than measure capability. None of it appears in a leaderboard, and all of it is why a buyer should ask for the acceptance record instead of the headline number.
04What a buyer should ask for
Private holdout discipline, contamination screening, task versioning, verifier hardening against reward hacking, and failure logs with attempt budgets. If a vendor can only show clean outputs and a public score, you are looking at the billboard.
About DataMining Founded by PhD researchers from the Institute of Software, School of Computer Science, Peking University, DataMining builds original task data and executable evaluation environments for frontier models and agents, with reproducible rubric specifications and bootable Harbor Task verifiers. Benchmarks named here are public third-party benchmarks; we align capability requirements without copying tasks. Visit datamining.cloud or write to contact@datamining.cloud.