What Enterprises Actually Buy: A Line-by-Line Breakdown of 16 Benchmark Data Asset Lines
Procurement teams receive vendor materials every day, and the most common pitch line is "we can do XX benchmark same-style." What sits behind that sentence varies enormously: under the same "Deep SWE same-style" label you may get a batch of item files, or the full bundle of items, environment, reference solutions, and verifiers — or a delivery whose scoring scripts cannot even run. Buy the wrong shape and the data never plugs into your training or evaluation pipelines; the budget goes to things you cannot use.
This article takes "same-style" apart. Our public capability catalog holds 16 asset lines: 10 mapped to mainstream public benchmarks, 4 grown out of real customer scenarios, 2 standardized runtime-environment product lines. For each line we spell out four things: which benchmark it tracks, what the official specs say, what deliverable shape the buyer receives, and what acceptance looks like.
We are a vendor ourselves, and this catalog is our shelf; dissecting our own shelf invites the charge of self-promotion. So each line also carries acceptance checks written from the buyer's side — checks that apply to every vendor, us included.
01The short answer
The 16 lines fall into six groups: hard-science exam questions (HLE Math, Physics, Chemistry — 3 lines), runtime environments (Windows and Ubuntu standardized VM images — 2 lines), coding engineering (Frontier-Eng, Deep SWE, Frontier SWE — 3 lines), general agents (ALE — 1 line), embodied AI (grounding QA, grounding annotation, navigation SFT, DAgger post-training — 4 lines), and long-horizon agents (Automation Bench, Toolathlon, Harvey LAB — 3 lines).
Enterprise money actually goes to three kinds of products. The first is evaluation tasks: items, reference solutions, and re-runnable verifiers delivered as a package, used to measure model capability. The second is training data: SFT corpora, annotated frames, DAgger correction data, used to feed training. The third is runtime environments: standardized images and containers, the foundation that lets the first two be reused over and over. When a quote carries the same benchmark name, first establish which of the three it is, then talk price.
Each line also carries a difficulty-tier definition: L1 means similar to the original dataset — specs and evaluation criteria aligned, task content original; L2 means further improvement on top of the original dataset; L3 means fully beyond the original dataset, newly authored in its style. The tier is not a marketing word; each line below states the evidence behind it.
021. Hard-science exam questions: the three HLE lines
HLE (Humanity's Last Exam), organized by the Center for AI Safety (CAIS) together with Scale AI and built with domain experts worldwide, is a public benchmark of very hard multimodal academic questions. We track it with three independent product lines: HLE-Math, HLE-Physics, and HLE-Chemistry, newly authored in HLE style by a competition-coach team, strictly multimodal, with an automated production pipeline built on top of human authoring. Difficulty tier: L3. Delivery scale: 2,000, 2,000, and 1,000 items respectively, with uncapped capacity and 3 working days to deliver.
What buyers receive is more than items. Each record is structured as: question, answer, worked solution, options, question images, option images, Pass@4 evaluation results, and four rounds of model dialogue with the corresponding judgments. QC is split by subject: math goes through formal-solver validation (Z3/SymPy), layered with multi-model cross-validation and double-blind expert re-review; physics and chemistry go through multi-model cross-validation plus double-blind expert re-review. Objectivity of judgment comes from programs and cross-review, not from a single annotator.
For difficulty evidence, one set of internal measurements: in our internal controlled evaluation on the HLE direction, these newly authored math, physics, and chemistry expert items lifted Seed-2.1-pro-0915's accuracy by 4.6, 5.3, and 2.7 percentage points respectively. The setup must be stated clearly: this is our internal controlled evaluation result; the model has no official HLE leaderboard entry; these numbers only indicate the difficulty range of the items and are not for any leaderboard comparison.
Acceptance checks: ask the vendor for three sample items and verify three things — whether the prompt is self-contained (figures, reference objects, and coordinate conventions all inside the prompt), whether the answer is uniquely decidable, and whether the solution gives a derivation chain rather than a bare conclusion. Miss any one and the item becomes a disputed item once it enters your evaluation set.
032. Runtime environments: Windows / Ubuntu standardized VM images
These two lines sell not items but environments. Training and evaluating desktop-operation agents (GUI agents) both need a clean, resettable, clearly specified Windows or Ubuntu environment, in which the model learns UI understanding, mouse and keyboard operation, multi-step task execution, software use, and exception handling.
Our image specs are hard-coded into the delivery documents: Windows prefers 11 Enterprise, then 10 Enterprise; Ubuntu prefers 24.04, then 22.04. 60G per image with at least 20G of free space kept, no split VMDK, BIOS boot, fixed naming rules. Operation data inside the environment is produced by real humans completing the tasks, containing task instructions, UI states, and operation trajectories — not script-synthesized. 7 working days to deliver.
Acceptance checks: can the image be restored to factory state and replayed repeatedly; are the trajectory fields complete (instructions, UI states, action sequences, timestamps); are the VM version and resource configuration written into the contract. The value of GUI data depends heavily on environment standardization — a vague environment means trajectories cannot be reproduced.
043. Three coding engineering lines: Frontier-Eng, Deep SWE, Frontier SWE
Coding is the most hotly contested group in evaluation data, and the three lines measure three different capabilities.
Frontier-Eng (reference benchmark: github.com/Einsia/Frontier-Engineering) is a generative-optimization benchmark of real engineering tasks: 47 tasks across five domains — computing and quantum information, operations research and decision science, robotics, control and energy systems, optics and communication systems, and physical science and engineering design. The model repeatedly generates and revises executable code within a fixed interaction budget; scores come from domain verifiers that compile and execute the code against target values and constraints, where each task's gold/silver/bronze baselines are frozen from the top-three best-feasible scores of the v1 snapshot, awarding 1.00, 0.67, and 0.33 per tier. We deliver tasks of the same kind to the same specs, difficulty tier L1. This line has one verifiable result: after training with our Frontier-Eng-style task data, GLM-5 reached third place on the benchmark's official leaderboard.
Deep SWE (reference benchmark: built by Datacurve, 113 original tasks, 91 active open-source repositories, 5 programming languages) measures repository-level long-horizon modification: the model independently understands the codebase, designs an approach, completes the modification, and runs the tests. At evaluation time, submissions are extracted as patches and replayed for verification in clean containers; the metrics are pass@1 and pass@4 (the official leaderboard shows pass@1; pass@4 is reported in the paper). We deliver tasks of the same kind to the same specs, 200 items/week, QC via multi-model cross-validation plus double-blind expert re-review. Two comparable measured numbers exist on this line: in Deep SWE-class task data comparisons, GPT-5.6-sol scores about 15 percentage points lower in accuracy on our data than on comparable data from other vendors in the same category, and agent reasoning trajectories run about 20 turns longer. Both numbers say the same thing: the tasks push the model onto longer reasoning paths. In our own difficulty calibration records, GPT-5.6-sol (xhigh) succeeded on 3/20 attempts across 5 DeepSWE-class repository tasks — the task difficulty and the model tier are both on record here.
Frontier SWE (reference benchmark: built by Proximal, v2 contains 34 tasks) measures very long-horizon programming: tasks span five categories — systems implementation, scientific computing, performance optimization, visual reasoning, and AI research; each task provides a complete Linux environment and a 20-hour execution budget; each item runs 5 independent trials averaged as mean@5; and final artifacts are assessed by deterministic automated verifiers. We deliver tasks of the same kind to the same specs, difficulty tier L1.
Acceptance checks: the three lines share one thing in common — "the verifier can be re-run." Ask the vendor to demonstrate one full evaluation: take a submission, extract the patch, replay it in a clean container, output the score. If they cannot demonstrate it, the difficulty claim is just words.
05A mid-article case: audit numbers inside a DeepSWE-class delivery package
A DeepSWE-class task package we delivered in October 2026 achieved item-by-item auditable delivery discipline: a full SHA256SUMS hash manifest; a PROVENANCE file recording the upstream dataset commit, the official grader hash, and the base commits of 5 repositories; validation.json recording 5 tasks, 33 trajectory slots, and 27 valid trajectories — with 6 historically invalid runs kept as null rather than deleted, 3 empty patches honestly labeled, apply_failed at 0, and full CRC checks passed. The meaning of these numbers sits on the failure side: failed trajectories and empty patches are neither deleted nor rewritten, so the accepting party can verify the true result of every single run. A vendor that ships only successful samples is delivering half the evidence.
064. General agents: ALE (Agents' Last Exam)
ALE, co-organized by Berkeley RDI and developed with 250+ industry experts, covers 55 subfields under 13 industry clusters — 960 expert task workflows and 1,490 task instances per the paper (122 active tasks remain in the official repo at v1.1 after 30 retirements). Its distinguishing feature is testing agents with industry-expert tasks drawn from real practice, not padding the count with synthetic office tasks.
How we deliver ALE-class tasks: produced by experts, using real, license-permitted professional cases; each item ships with an executable reference solution and an independent scorer; delivered in Harbor Task format, each item with its own independently bootable Docker environment, and each record with a review oracle. Difficulty tier L2 (improvement on top of the original specs); capacity 200 items/week; 7 working days.
On the demand side, for comparison: industry ALE procurement requirements have already written "expert profiles across 15 domains," "a difficulty bar of at least one non-perfect score in pass@3," and "delivery of an engineering structure package plus three independent run trajectory packages" into their terms. Evaluation-data procurement is shifting from buying item lists to buying capability specs with acceptance clauses.
Acceptance checks: can each item's environment boot independently; is the scorer a standalone program; sample three items and verify whether the reference answers can be re-derived from the prompts and materials. The biggest risk in industry-expert tasks is this: the expert writes a conclusion, but no one can verify how the conclusion was reached.
075. Four embodied AI lines: from real-robot data to the DAgger feedback loop
All four embodied lines grew out of real customer projects; the data source is real robots, not synthesis. The customer supplies real-robot data, we do the processing, annotation, and structuring, and the whole package is delivered in 45 working days.
Grounding QA (30k items): question-answer pairs built from robot trajectories collected in real scenes — 2D projection coordinates, bbox projection coordinates, scene captions, proprioception, and motion perception — used to improve multimodal models' grounding capability in real robot scenarios. Grounding annotation (300k frames): precise bbox and mask annotation of manipulation targets, with double-blind expert re-review. Both lines were adopted by the same leading embodied-intelligence vendor.
Embodied navigation SFT (1M items): training data for multimodal LLMs' decision-making and spatial perception on embodied navigation tasks. DAgger post-training data (800k items): for robustness and error-correction capability on navigation tasks. Both lines were adopted by a university lab.
The line worth expanding here is DAgger. It is itself a feedback-loop mechanism: let the model run first, collect the states where it drifted, re-annotate them, and keep training on the new data. Taken together, the four lines cover a complete embodied data stack — "perception annotation → QA understanding → imitation learning → error-correction reinforcement" — which is also the staged order in which enterprise embodied teams buy.
Acceptance checks: are the provenance and collection conditions of the real-robot data traceable; how are bbox and mask edge cases defined in the annotation spec; what is the sampling rate of the double-blind re-review. Quality problems in embodied data mostly hide in the definitions of edge cases.
086. Three long-horizon agent lines: Automation Bench, Toolathlon, Harvey LAB
Automation Bench (reference benchmark: released by Zapier, github.com/zapier/AutomationBench) has an official set of 600 tasks, 100 per domain, plus 200 simple starter tasks that do not count toward official scores, covering 47 simulated SaaS applications. Each task initializes a simulated enterprise environment; after the agent completes the business flow autonomously, program assertions check the final state. We deliver tasks of the same kind to the same specs, 200 items/week. We dry-ran three demo tasks with gpt-5.6-sol (high) and kept full trajectories: attempt-01 averaged 0.63, and per-task assertions ranged from a full 14/14 pass to just 2/20 — program assertions pull task difficulty apart widely.
Toolathlon (reference benchmark: arxiv 2510.25726) has 108 tasks, 32 software applications, and 604 tools, averaging roughly 20 interaction turns; it emphasizes diverse applications, realistic initial environment states, and long-horizon workflows, with a dedicated evaluation script for each task's execution outcome. We deliver tasks of the same kind to the same specs, 200 items/week.
Harvey LAB (reference benchmark: introduced in Harvey's official launch post) has 1,250 tasks across 24 legal practice areas with over 75,000 expert-written rubric criteria; each task contains work instructions, client matter materials, and deliverable requirements, simulating a law-firm partner assigning work and lawyers processing materials and submitting deliverables. We deliver tasks of the same kind, and calibrate difficulty against an explicit objective function: for our Harvey LAB-style legal compliance review items, the difficulty floor is set so that gpt-5.6-sol (high) fails at least 2 of 4 runs.
Acceptance checks: is the application state in the simulated environment fully initialized; are the assertion or evaluation scripts delivered at the same version as the tasks; does every rubric criterion cite its source. Long-horizon tasks carry their cost in the environment and their risk in the scoring — check both.
09How to set acceptance criteria when buying an evaluation set
Having walked through the 16 lines, the acceptance criteria converge to four.
First, difficulty claims must carry three elements: on which model, compared against what, and on which metric. "Our tasks are harder" without these three elements is marketing. Second, failure data must be kept: failed trajectories, invalid runs, and rejected items are the evidence for difficulty calibration; a delivery of only successful samples is half the evidence. Third, judgment must be re-runnable: program verifiers or item-by-item checkable criteria — a model judge's impression score does not enter acceptance. Fourth, tiers must be written into the contract: L1/L2/L3 correspond to different evidence — L3 requires the authoring records of brand-new items, L1 requires a spec-alignment statement against the original benchmark.
One failure self-report of our own, as a demonstration of the practice: on our Terminal-Bench-Science-style scientific computing items, DeepSeek-V4-Flash-Vision-Exp exhausted its budget on all 5 trials per item (80-step cap, 1800 seconds, an offline container, temperature mixed between 0 and 0.6); computed by the binomial distribution, the one-sided 95% confidence upper bound on the success rate is 45.1%. The original wording of the verdict: "Observed failure under the recorded model, agent, tools and budget only; not universal inability." Failures need a statistical setup; conclusions need a stated scope — this is the standard practice in our delivery packages, and it should be the standard requirement at acceptance.
10A checklist to self-check against
| What you are buying | Check item | Failure signature |
|---|---|---|
| Evaluation tasks | Do they include reference solutions and re-runnable verifiers | Item files only |
| Evaluation tasks | Do difficulty claims carry model name, comparison target, and metric name | Only "ours are harder" |
| Evaluation tasks | Have items been screened for search-solvability | Solvable by a web search |
| Training data | Is the provenance real (real robots, real work) | Fully synthetic with no provenance statement |
| Training data | Does annotation have double-blind or cross review | Delivered after single-annotator labeling |
| Training data | Are failure and correction samples kept | Successful samples only |
| Runtime environments | Can images or containers be reset and replayed | One-shot snapshots |
| Runtime environments | Are specs (version, resources, boot method) documented | Verbal descriptions |
| Delivery discipline | Is there a hash manifest and provenance record | A folder plus a README |
| Delivery discipline | Are invalid runs kept as they are | Failures deleted, successes only |
| Delivery discipline | Is QC independent of the item author | Expert self-review is the final gate |
| Procurement action | Run a small POC plus a held-out test set first | One big order betting on self-reported metrics |
11Summary
16 lines, three products: evaluation tasks buy "capability measurement," training data buys "learnable signal," and runtime environments buy "reusable grounds." When the same benchmark name appears on quotes, first tell which of the three it is, then look at the difficulty tier and the acceptance evidence.
The four acceptance criteria apply to all vendors: difficulty claims with three elements, failure data kept, re-runnable judgment, and tiers written into the contract. For a quote that fails all four, decline it no matter how low the price.
We are a vendor too. Sample data, acceptance records, and QC reports for all 16 lines can be inspected one by one — you are welcome to verify us against the checklist above.
12Related work we have delivered
| Case | What we did | Scale | Turnaround |
|---|---|---|---|
| HLE three subjects (Math/Physics/Chemistry) | Newly authored by a competition-coach team; math via Z3/SymPy formal validation, physics and chemistry via multi-model cross-validation plus double-blind re-review | 1,000–2,000 items per subject | 3 working days |
| DeepSWE-class delivery package | Real repository tasks with full-chain audit via SHA256SUMS, PROVENANCE, and validation.json; failed runs kept as they are | 5 tasks, 33 trajectory slots, fully recorded | Per project |
| Three scientific computing items (Terminal-Bench-Science style) | 3 screened from 8 candidates, all 5 trials per item recorded, interactive trajectory replay viewer | All 5 trials per item recorded | Per project |
| ALE-class expert tasks | Expert-produced, per-item Docker environments and independent scorers, Harbor format | 200 items/week | 7 working days |
| Four embodied AI lines | Real-robot grounding QA and annotation, navigation SFT and DAgger post-training | 30k items / 300k frames / 1M items / 800k items | 45 working days |
| Industry expert task packages | Practicing-expert authorship in healthcare, legal, and finance, with distractor files, evidence indexes, and Rubrics; contradiction QC before delivery | Customized per industry and project | Per project agreement |
What we do: DataMining (datamining.cloud) builds original task data for LLM and agent R&D. Founded by a PhD team from the Institute of Software, School of Computer Science at Peking University, we cover coding, complex reasoning, multimodal, and agent tasks, constructing original same-kind tasks to the capability bars of ALE-Bench, Frontier-Eng, HLE, Frontier SWE, Deep SWE, Program Bench, APEX Agents, and peer benchmarks. Coding data is our core: on Deep SWE-class tasks, GPT-5.6-sol scores about 15 percentage points lower in accuracy on our data than on comparable data from other vendors in the same category, with agent reasoning trajectories about 20 turns longer.
For sample data, acceptance records for the 16 lines, or a custom proposal, contact the team: contact@datamining.cloud
13Further reading
- Four Dimensions for Evaluating AI Data Vendors: Expert Network, Measurement Science, Execution Environments, Feedback Loops
- Leaderboards locked in a vault: the data-contamination technology chain and acceptance rules for the private-evaluation era
- Open-source exam rooms, pricier sandboxes: the RL-environment bill gets re-split
- What enterprises procure has changed: from buying datasets to buying environments and verification capability
- A $2.5B bet on one exam room: China's data-vendor table is set