Four Dimensions for Evaluating AI Data Vendors: Expert Network, Measurement Science, Execution Environments, Feedback Loops

The most common claims in vendor pitches are: tens of thousands of registered experts, datasets with hundreds of thousands of items, and accuracy lifts reported by customers. All three are self-reported. The four dimensions cut closer to the business because they answer one question: where do capacity and quality actually come from.

We are a vendor ourselves, so the position is stated up front: every evidence standard below is also applied to us, and each dimension states what we can and cannot show.

01The short answer

Each dimension has one kind of evidence to check. Expert network: look at authoring quality — do experts write the tasks themselves, can reference solutions be traced back to source material, is there independent QC before delivery? Registration counts are not evidence. Measurement science: look at whether judgments can be re-run — correct/incorrect decided by a program or item-by-item criteria, and difficulty claims backed by measured comparisons under a stated setup; item counts are not evidence. Execution environments: look at the delivery shape — a full bundle of task + environment + reference solution + verifier, or just a sample file; whether environments reset and replay cleanly is the dividing line for training integration. Feedback loops: look at task provenance — real work or invented tasks, and whether failure data can be collected back for the next training round. One procurement action covers all dimensions: run a small POC first, with a held-out test set the buyer controls.

02Dimension 1: Expert network — authoring quality, not headcount

The easiest number to compare is expert scale: tens of thousands of registered experts across hundreds of industries. But registration and capacity are different things; how many qualified items a registered expert actually produces per month is the real capacity metric, and only the vendor sees it. So do not compare scale. Check three things.

First, what experts do in the pipeline. If experts only grade model-generated items, the bottleneck is the model, not the experts; if experts author tasks, write reference solutions, and define rubrics themselves, the bottleneck is professional judgment. The Agents' Last Exam (ALE) benchmark is a public reference: co-led by Berkeley RDI, developed with 250+ industry experts, covering 55 subfields across 13 industry clusters, with 1,000+ tasks collected per the paper — tasks that practitioners bring from real work. Harvey's Legal Agent Benchmark is another: 1,250 tasks across 24 legal practice areas with over 75,000 expert-written rubric criteria. The expert deliverable is the rubric, not just an answer — that is the difference between experts who author and experts who lend their names.

Second, traceability of reference solutions. Every conclusion in an expert-written reference solution should point back to "which document, which passage". Without that, QC cannot tell derived conclusions from embellished ones. In our industry expert task packages, the delivery structure always includes one field: the expert's solution rationale with an evidence index.

Third, independent QC before delivery. Experts reviewing their own work catch limited errors. We run a contradiction QC on every expert-delivered task package, checking five dimensions: prompt completeness, file and identifier consistency, correctness and derivability of reference deliverables, cross-file consistency, and rubric quality. One real example: in a food-safety epidemiology task package, the source material never stated when the implicated dinner began, but the reference solution assumed 18:00, computed an incubation window from it, and ranked pathogen priorities accordingly; the material recorded only abdominal pain and diarrhea, yet the solution asserted negative symptoms such as "no frequent severe vomiting". Both issues would change the professional conclusion. The QC report flagged them as high risk and explicitly recommended not submitting, rather than issuing a soft warning.

Our boundary on this dimension: we are a small authoring-first expert network — core task authorship is done by a PhD team from the Institute of Software, School of Computer Science at Peking University, with industry tasks authored by practicing experts in healthcare, legal, and finance. We are not a platform whose strength is registered headcount. Buyers seeking mass generic annotation capacity are comparing a different axis entirely.

03Dimension 2: Measurement science — reproducible judgment is the moat

This dimension converts real, long-horizon, high-stakes work into auditable, reproducible data with fine-grained scoring criteria. Translation ability is scarcer than domain know-how, because it requires both business understanding and evaluation engineering. Check three things.

First, can the judgment be re-run. "Correct" should be decided by a program or by item-by-item checkable criteria, not by a model judge's impression. Frontier-Eng is a clean reference: 47 real engineering tasks across five domains (computing and quantum information, operations research and decision science, robotics and control, optics and communication systems, physical science and engineering design), where the model iteratively edits code within a fixed interaction budget and scores come from domain verifiers that compile and execute against frozen thresholds of 1.00, 0.67, and 0.33. Scores are run, not judged.

Second, has the task set itself been measured. HLE (Humanity's Last Exam) screens its question bank with search-equipped models and then audits items one by one, removing anything a search engine can solve — so the benchmark measures reasoning, not recall. That layer of QC on the tasks themselves is the difference between scores that reflect capability and scores that reflect contamination.

Third, is the difficulty claim measured or just asserted. "Our tasks are harder" only means something as a measured comparison on an agreed model, stating the model name, the comparison target, and the metric. Any difficulty claim missing these three elements should be treated as marketing. For HLE-style items we can contribute one internal measurement: on math, physics, and chemistry expert items newly authored by a competition-coach team, Seed-2.1-pro-0915 gained 4.6, 5.3, and 2.7 accuracy points respectively in our internal controlled evaluation. The setup matters: this is an internal controlled evaluation — the model has no official HLE leaderboard entry — so the numbers indicate the difficulty range of the items, not leaderboard standings. Math items are validated with formal solvers (Z3/SymPy); physics and chemistry go through multi-model cross-validation plus blind expert re-review.

Our own difficulty claim is in the next section.

04A worked case: two measured numbers on Deep SWE-class tasks

Our Deep SWE-class tasks follow the original recipe: live open-source repositories, real functional verifiers, real reference solutions; submissions are extracted as patches and replayed in clean containers. Delivery capacity is 200 items/week with a 7-working-day turnaround, QC via multi-model cross-validation plus blind expert re-review. The comparable difference is two measured numbers: in Deep SWE-class task comparisons, GPT-5.6-sol scores about 15 percentage points lower in accuracy (ACC) on our data than on comparable data from other vendors in the same category; on the same tasks, agent reasoning trajectories run about 20 turns longer. Both numbers say the same thing: the tasks push the model onto longer reasoning paths instead of letting it turn in the exam early. For teams training long-horizon coding ability, trajectory length is itself trainable signal.

05Dimension 3: Execution environments — a bundle, not a sample

Tasks are only one piece of the deliverable. To plug data into training or evaluation, buyers need the full bundle of task + environment + reference solution + verifier — and the environment is the dividing line, because it decides whether the data can be reused at scale. Check three things.

First, can environments reset and replay. Whether evaluation runs once or training runs ten thousand times, the environment must return to its initial state. Terminal-Bench attaches per-task verifiers under the Harbor framework; when we deliver ALE-class tasks, each task ships with an independently bootable Docker environment, an independent scorer, and a per-item review oracle in Harbor task format. For virtual-machine tasks we deliver standardized images: Windows 11 Enterprise or Ubuntu 24.04, 60G per image, with desktop-operation data produced by real humans completing tasks inside the image — task instructions, UI states, and operation trajectories — in 7 working days.

Second, is reward hacking prevented. If a verification environment ships with answer files or leftover state from previous runs, the model can learn cheating paths. DeepSWE extracts submissions as patches and replays them in clean containers; FrontierSWE's verifier restarts from a fixed image with a clean filesystem. Two direct questions for any vendor: how is the verification environment kept clean, and do timeouts and context overruns count as failures.

Third, is capacity a pipeline. Environment production pays off in batch: our mainstream delivery capacity for coding and agent tasks is 200 items per week with a 7-working-day turnaround. That number depends on standardized environments — without resettable environments it does not hold.

The Terminal-Bench 3.0-to-4.0 update is worth studying: 74 tasks cut to 66, with 8 removed (2 saturated, 2 model refusals, 2 publicly solved, 2 with quality or platform-compatibility issues), 19 fixed, and an explicit breaking-change notice requiring 3.0 scores to be re-run on 4.0. Public benchmarks cycle their own item pools; vendor-delivered task data needs the same re-verification mechanism. Ask one question: how often is the task bank re-verified, and what happens to items that fail.

06Dimension 4: Feedback loops — where tasks come from, and whether failures return

This dimension sets the ceiling. Invented tasks can test format-following, not the complexity of real work; real tasks come from two sources — customer scenarios and original re-creations alongside public benchmarks. Feedback loops are a different question: after the model fails on these tasks, can the failure site be collected back into the next round of training data.

Our strongest evidence here is embodied AI. A leading embodied-intelligence vendor supplied real-robot data for grounding QA (30k items: robot trajectories, 2D projections, bbox projections, scene captions, proprioception and motion perception) and grounding annotation (300k frames of precise bbox and mask labels); for a university lab's navigation project we delivered 1M multimodal SFT items and 800k DAgger post-training items in 45 working days. DAgger itself is a loop mechanism: let the model run, collect the states where it drifted, re-annotate, and continue training on the new data. The tasks come from real robots, not synthesis.

Our boundary here is equally clear: we do not operate a product that continuously generates feedback data; loops happen inside customer projects. What we can show is real task provenance plus a working loop method; what we cannot show is a self-operated scenario producing data continuously. If this dimension is a hard requirement for you, the right verification is not listening to any vendor's loop story — it is reviewing the provenance material behind the tasks and checking, inside the POC, how close the tasks sit to your real workflow.

07Grading the evidence, and the procurement action

Across all four dimensions, what vendors provide falls into three tiers. Tier one, self-reported metrics: registration counts, dataset totals, vague customer references — leads, not evidence. Tier two, inspectable artifacts: sample tasks, reference solutions with evidence indexes, QC reports, verifier code, environment images — checkable item by item. Tier three, reproducible measurements: comparisons run on a model the buyer designates or accepts, under an agreed setup — the only tier that cannot be produced by storytelling.

The matching procurement actions are converging across the industry: a small real POC, a held-out test set kept by the buyer, and a review of data provenance and licensing. We welcome all of them, because reproducible evidence favors us — which is exactly why we can publish this checklist.

08A checklist you can use directly

Dimension	Check item	Failure signature
Expert network	Do experts author tasks, reference solutions, and rubrics themselves	Experts only grade model output
Expert network	Can every conclusion in reference solutions be traced to source material	QC cannot tell derivation from embellishment
Expert network	Is there QC independent of the author before delivery	Author self-review is the final gate
Measurement science	Is correctness decided by a program or item-by-item criteria	Model-judge impressions
Measurement science	Do difficulty claims carry model, comparison target, and metric	Only "ours is harder"
Measurement science	Has the item bank been screened for search-solvable items	Scores mix recall and contamination
Execution environments	Does every task ship with a bootable environment and independent verifier	A sample file is the deliverable
Execution environments	Is the verification environment clean; do timeouts fail	Cheating paths exist
Execution environments	Is the capacity number sustainable	One-off delivery, no continuation
Feedback loops	Do tasks come from real work	Invented tasks
Feedback loops	Can failure data be collected back into training	Failure sites are discarded
Procurement action	Small POC plus held-out test set before scale	One big bet on self-reported metrics

09Summary

The four dimensions split into two pairs. Expert network and measurement science decide whether the tasks are good: authoring quality sets the ceiling, reproducible judgment decides whether scores mean anything. Execution environments and feedback loops decide whether the data scales: environments make it pluggable into training, real provenance and loops decide whether the dataset keeps growing.

Per dimension, ask the vendor for one thing: authoring samples and QC reports for the expert network; measured comparisons under a stated setup for measurement science; an environment dry-run for execution environments; task provenance documentation for feedback loops. Negotiate volume after all four are on the table; for anything missing, verify with a POC and a held-out test set.

We are a vendor too — every standard in this checklist is one we invite you to hold us to.

10Related work we have delivered

Case	What we did	Scale	Turnaround
HLE-Math / Physics / Chemistry	Strictly multimodal expert items newly authored by a competition-coach team; formal-solver validation for math, multi-model cross-validation plus blind re-review for physics and chemistry	1,000–2,000 items per subject	3 working days
Deep SWE-class coding tasks	Live repositories, real functional verifiers and reference solutions, clean-container replay verification	200 items/week	7 working days
ALE (Agents' Last Exam)-class tasks	Expert-authored, per-task Docker environments and independent scorers, Harbor format	200 items/week	7 working days
Windows / Ubuntu VM images	Standardized images with real human desktop-operation trajectories	60G per image	7 working days
Embodied AI (four projects)	Grounding QA, bbox/mask annotation, navigation SFT, and DAgger post-training data on real-robot data	30k QA / 300k frames / 1M SFT / 800k DAgger	45 working days
Industry expert task packages	Practicing-expert authorship in healthcare, legal, and finance, with distractor files, reference deliverables, evidence indexes, rubrics, and contradiction QC	Per industry and project	Per project

DataMining (datamining.cloud) builds original task data for LLM and agent R&D — coding, complex reasoning, multimodal, and agent tasks — constructed to the capability bars of ALE-Bench, Frontier-Eng, HLE, Frontier SWE, Deep SWE, Program Bench, APEX Agents, and peer benchmarks. Coding data is our core: on Deep SWE-class tasks, GPT-5.6-sol scores about 15 percentage points lower in accuracy on our data than on comparable data from other vendors in the same category, with agent trajectories about 20 turns longer.

For sample data, an evaluation-set inventory, or a custom proposal: contact@datamining.cloud

11Further reading

• A $2.5B bet on one exam room: China's data-vendor table is set
• Open-source exam rooms, pricier sandboxes: the RL-environment bill gets re-split
• Anthropic cut off web access for internal evals — the reason is on their own blog: environments leak
• After one supply-chain attack, vendor security became a hard procurement requirement