Data Procurement Shifts from Datasets to Environments

01What the posting actually says

The duties: evaluate suppliers' data collection capability; evaluate data processing pipelines and quality control standards; in technical due diligence, examine the supplier's automation level, cost control, and iteration roadmap. No line asks how many annotators you have.

ByteDance has set up a first-level AI Data and Security department covering standard-setting, sourcing, synthesis and cleaning, and quality evaluation; procurement is one link inside it (智能涌现). Both point one way: procurement is becoming a technical evaluator.

Why: if procurement cannot read a verifier, acceptance passes to the algorithm team — hence technical buyers. My judgment: technical due diligence roles for environment contracts become standard in big-tech procurement within one to two years — unless next year's are still accepted by algorithm teams alone.

02What the money is doing

On July 9, Mercor acquired Deeptune, which trains AI agents (TechCrunch); the stated purpose: highly realistic software application training environments, fast (The Information). Its researchers hold that simulation environments beat manual annotation in many professional scenarios. Mercor's position: demand for human tuning still grows, and legal cases need human review — but a company buying environment capability does not plan to sell only labor.

Snorkel closed a $350 million Series E on September 22 at a $375 million annualized run rate, selling RL environments and ready-made datasets; expert payouts are booked as cost of goods sold. Mercor's gross annualized revenue is $2 billion, Handshake hit $1 billion earlier this year, and these companies pay 60-70% of revenue to experts. Not comparable: Snorkel's figure is net of expert costs, Mercor's is gross.

03The deliverable: five parts, one expensive

A dataset ships static samples; an environment ships five things, per Lee Han Chung's March breakdown: task set, harness (the model-environment interaction shell, with tools and sandbox), verifier, state, configuration.

The verifier is the most expensive part. Refining one rubric takes more than 10 hours; OpenAI's published medical evaluation example assigns close to 50,000 criteria across tasks, and labs order tens of thousands to hundreds of thousands of rubrics per training run (The Verge). Anthropic allocated over $1 billion for RL data in 2025 alone (智能涌现).

Environments get bought one domain at a time. Mercor's CEO Brendan Foody: reinforcement learning's generalization radius is limited, so labs build evaluations for every direction they optimize — a seller's view, matching Lee Han Chung's: environment diversity decides agent capability breadth more than any single environment's size. Surge AI's Edwin Chen counters that new and old training forms are nearly complementary — datasets will not disappear, but new budget flows to environments and verification.

Execution is commoditizing: Alibaba Cloud's commercial agent sandbox creates 100,000 sandbox instances per minute, with AgentRL listed. Once sandbox compute is a commodity, vendor value sits in tasks, verifiers, and quality — if top vendors price environment contracts by sandbox compute-hours, this judgment is wrong.

04Acceptance: four questions, three numbers

Dataset acceptance is sampling; environment acceptance asks four questions — solvable, bug-free, right difficulty, gameable. Three numbers show why these are hard (甲子光年):

• Tencent's Hunyuan team collected 47,678 public terminal environments from Hugging Face and GitHub; after filtering on quality, solvability, and difficulty, 127 survived — a 99.7% rejection rate. When task requirements and detection standards disagree, small models barely notice; large models hack the gap outright.
• OpenAI re-audited SWE-Bench Pro in July: about 30% of the 731 public tasks have problems — over-strict tests, incomplete prompts, insufficient test coverage.
• Anthropic held model, tasks, and harness fixed and changed only CPU and memory: Terminal-Bench 2.0 moved 6 percentage points between the tightest and loosest settings.

Environments are software to be audited: read verifier code, run pass-rate distributions, reproduce resource configurations. Vendor says the model is weak; buyer says the tasks are unsolvable. Only re-running settles it. Contract terms: resource configurations, verifier versions; buyer re-runs are the acceptance standard, vendor self-tests reference only.

The AutoEnv paper, cited by Lee Han Chung, puts average generation cost at about $4; the bottleneck is verifier quality — weak reward functions teach wrong behavior at scale. Where programmatic verification works, do not use an LLM as judge.

05Pricing and IP

Pricing: datasets price per item or per hour; environments have no natural unit. Price per environment that passes acceptance; the buyer pays sandbox compute directly — hourly pricing makes slow delivery profitable for the vendor.

IP: an environment contains a harness and verifiers — the vendor wants to resell it to many buyers, the buyer wants exclusivity. Large buyers already run data-exclusivity playbooks: exclusive windows on datasets, staged buyouts of a supplier's key people (智能涌现). Exclusivity windows and verifier ownership become the center of negotiation.

Owning environment capability, as Mercor now does, removes the risk of being unbundled or replaced in-house. If three or more top labs announce in-house environment teams before Q4, the bet is wrong.

Revenue accounting will split: staff-time sellers report gross; environment sellers report the number after expert costs. The Alibaba role pays 40-60K per month; its five duties are mapping, evaluation, due diligence, sourcing, and coordination — not one is placing an order.