What should a vendor deliver with a coding task dataset?

A complete package has five parts: task instructions, a resettable execution environment, grading tests and a verifier, a reference solution with an evidence trail, and an acceptance measurement record stating model version, attempt count, and environment. Hidden tests, answer keys, and golden patches are evaluation assets — verify their existence and isolation rather than requesting copies.

A vendor claims their tasks are harder. How do I verify that?

Require measured numbers with a stated budget: which model and reasoning tier, compared against what, using which metric — for example passes under a fixed model tier with four valid attempts per task. Difficulty claims without these three elements should be treated as marketing.

How do I check that a vendor's verifier will not pass reward-hacking solutions?

Three actions: ask whether the verifier stores the answer or re-derives it independently; ask for a live demo where a deliberately wrong solution is rejected; and check that grading starts from a clean state with submissions replayed as patches.

How to Accept a Coding / Agent Task-Data Delivery: A Buyer's Checklist

Q: How should a first engagement be structured?

A small real POC on your own business slice, a hidden test set you retain, batch-by-batch acceptance with failure trajectories kept as evidence, and contract terms listing the five deliverables with a re-verification cycle.

This checklist is written for LLM teams buying coding or agent task data: what a delivery should contain, what makes a measurement count, how to audit the grader itself, how to screen for contamination, and which traps recur. It is written from the buyer's side because a checklist only binds when the buyer runs it — a vendor-authored checklist is a promise, a buyer-run checklist is a process. We are a vendor; every item below is also one our own deliveries must pass.

01Summary first

Acceptance has three layers: inspect the artifacts (complete package, traceable provenance), run the measurement (budget-stated pass@k, clean grading environment), and audit the verifier itself (can it reject a reward-hacking solution?). Most disputes happen at the third layer — buyers check the tasks but not the ruler that grades them. Related reading: Four Dimensions for Evaluating AI Data Vendors (English / 中文).

02What a complete task package looks like

Start acceptance by counting the deliverables. A trainable, re-runnable package has five parts; miss one and the later layers have nothing to land on:

Deliverable	Purpose	Acceptance focus
Task instruction	The only input the model sees	Self-contained: required materials, constraints, and expected output are stated in the brief
Execution environment	Where the task happens	Rebuildable from definition (image / Dockerfile), reset-and-replay; network and tool permissions declared
Tests & verifier	Judges whether the task was solved	Judges final state, not prose; grading starts from a clean state
Reference solution	Proves solvability; calibrates difficulty	Comes with a reasoning trail and evidence index — every conclusion traces to input material
Acceptance record		Model version, attempt count, environment, and whether failure trajectories are retained

Also know what should not be handed over: hidden tests, answer keys, and golden patches are evaluation assets — handing them to anyone but the buyer destroys the task's evaluation value. At acceptance, verify their existence and isolation (tests read-only inside the grading environment, separated from the solution), not request copies. A public structure example of a real package lives in our GitHub examples repository (with runnable verifiers).

03Run the measurement: pass@k needs a stated budget

"Our tasks are harder" is only meaningful as a measured number on an agreed model, stating three elements: which model (and tier), compared against what, using which metric. Require an acceptance record under a uniform budget — fixed model and reasoning tier, four valid attempts per task, same execution environment and grading. Numbers without a budget do not enter acceptance. Our own standard, including its limitations statement (failure under the recorded models, tools and budget — not a claim of general inability), is the Frontier Task Difficulty Report 2026.

04Audit the verifier: three actions against reward hacking

Great tasks with a loose grader produce noise. Three executable actions:

Ask: does the verifier store the expected answer or re-derive it? Stored-answer graders are open to answer extraction and memorization; re-deriving graders (deriving ground truth through an independent route and comparing) resist both.

Test: ask the vendor to run a deliberately wrong solution through the verifier and get rejected live; then a common reward hack (e.g., hard-coding outputs visible in the tests) — also rejected. A verifier that only lets the reference solution pass is judging something other than task-solving.

Inspect the environment: grading must start from a clean state, submissions replayed as patches in clean containers, test files read-only to the model. A grading environment with leftover answer files or previous run state teaches the model the cheat path, not the capability.

05Contamination screening: does the task already exist in public?

Contamination kills evaluation validity. Three checks: search the scenario's core terms plus the solution key against public benchmarks, repositories, and academic sets — a hit means re-synthesize; inspect the solution key itself — if the answer is a famous constant, a classic puzzle, or an actual repo bug, change it; require the vendor to state what their contamination screening covered (which libraries searched, what was changed). A task library without screening records counts as unscreened.

06Contradiction QA: a five-dimension framework buyers can reuse

Every industry-expert package we deliver passes a pre-delivery QA with five dimensions; buyers can reuse them directly as a re-check framework:

① Prompt completeness — everything needed to solve is present and unambiguous; ② file & identifier consistency — names, paths, and IDs match across files; ③ reference-deliverable correctness & derivability — the reference answer follows from the materials, every claim traced to a source; ④ cross-file consistency — no file contradicts another; ⑤ rubric quality — gradable as written. A real example we adjudicated: a food-safety epidemiology task whose materials never recorded the dinner time, while the reference answer assumed 18:00 to compute an incubation period — an error that changes professional conclusions. QA must flag it high-risk and state "do not submit", not merely note it.

07Acceptance traps

Difficulty claims without a budget — treat as marketing. Public benchmarks resold as original — re skinned SWE-bench-style items; check sources with the contamination method. Demo ≠ production quality — ten hand-polished samples can differ from production batches; contract a batch sampling rate and rework rule. Non-reproducible environments — environments depending on undocumented host state cannot reset; verify reset-and-replay before wiring into training.

08Structuring the POC

For a first engagement: a small real POC (dozens of tasks on your actual business slice) → a hidden test set you retain → batch acceptance with failure trajectories kept as evidence → contract terms listing the five deliverables, a re-verification cycle, and consequences for failure. The checklist itself is negotiation material: write sections 02, 04, and 06 into the contract and they bind harder than any verbal promise.

09Related material

Delivered datasets and RL environments: the datasets page; difficulty standard and limitations: the Frontier Task Difficulty Report 2026; runnable verifier examples: GitHub: datamining-ai/eval-task-examples. To run a POC against this checklist: contact@datamining.cloud.