Frontier Task Difficulty Report 2026: can difficulty be measured?
前沿任务难度报告 2026
Task difficulty is the core parameter of the training-data business, yet it is rarely delivered as a number. This report explains how DataMining designs, verifies, and measures tasks that frontier models have not solved yet, and publishes the full measurement caveat of one internal comparison: under the same evaluation pipeline, GPT-5.6-sol scores about 15 percentage points lower ACC on our Deep SWE-style coding tasks than on comparable data from another vendor, and its agents take about 20 more interaction rounds per task.
01Why difficulty needs measuring
Benchmarks saturate faster than procurement cycles. SWE-bench was a hard gate for engineering ability when released in 2023; its verified subset was largely cleared by mainstream models soon after. Humanity's Last Exam (Center for AI Safety and Scale AI, early 2025) collected expert questions from every field and saw rapid frontier progress within a year. The window between "publish a benchmark" and "saturate a benchmark" is now shorter than most data procurement contracts.
This creates a concrete buying question: when the tasks you purchase arrive, can the model already solve them? Tasks inside the model's ability boundary contribute near-zero gradient signal in RL training and overstate capability in evaluation. Difficulty is not a pricing extra; it is the precondition for data usefulness.
02How DataMining designs tasks
Every task is originally authored — nothing is rewritten from public question banks. Design follows three constraints:
- Calibrated to the ability boundary. Tasks are designed against the capability profile of public benchmarks: coding tasks align with Frontier SWE, Deep SWE, Frontier-Eng, and Program Bench; general agent tasks with APEX Agents and Agents' Last Exam (ALE, Berkeley RDI); algorithmic engineering with ALE-Bench (Sakana AI and AtCoder); expert academic problems with HLE. We match difficulty bands and capability dimensions, never the questions themselves.
- Contamination-resistant authoring. Problems are written by domain and engineering experts without referencing online sources. Before delivery, multi-model probes run across model families; any item showing overfit-style scores is rebuilt.
- Executable verification. Every coding and agent task ships with a runnable verification environment: unit tests, execution scripts, and decision rules. Evaluation results are reproducible without human grading.
Difficulty uses an internal grading scheme: L1 matches the reference benchmark; L2 goes one band deeper; L3 meaningfully exceeds it (for example, HLE-style expert problems).
03Internal comparison: caveat and results
We run periodic same-pipeline comparisons to calibrate difficulty. The most recent one:
- Model: GPT-5.6-sol (single version, single inference configuration);
- Tasks: Deep SWE-style repository-level coding tasks — one set of DataMining originals, one set of comparable tasks from another data vendor;
- Pipeline: identical execution environment, identical unit-test grading, identical agent scaffold and interaction-round cap.
Result one — accuracy. GPT-5.6-sol scores about 15 percentage points lower ACC on DataMining tasks than on the comparable vendor set. Under equal pipeline conditions, the ACC gap directly measures discrimination: the model fails more often on harder tasks.
Result two — trajectory length. Agents average about 20 more interaction rounds on DataMining tasks. Trajectory length is a behavioral indicator of how demanding a task is: the harder the task, the more act–observe–revise loops the model needs before converging or giving up. For RL training, longer trajectories mean denser policy-gradient signal; for evaluation, a wider discrimination range.
Honest caveats. These numbers come from an internal comparison, not a published third-party benchmark. The comparison target is "comparable tasks from another vendor" and does not name any company. The measurement covers one model version and one task domain (coding); we do not extrapolate it to other models or capability areas. Customers can request the full measurement methodology during evaluation.
04QA pipeline
Tasks pass layered QA before delivery, combined by data type:
- Formal solver verification — math answers and derivations checked with Z3 and SymPy;
- Multi-model cross-validation — several models answer and review independently; inconsistent items go to re-review;
- Double-blind expert review — domain experts audit item quality and answer correctness without knowing the author;
- Unit-test execution — coding and engineering tasks must run locally and grade deterministically;
- Contradiction QA (industry expert task packs) — five-dimension check over prompt completeness, file/identifier consistency, reference-deliverable correctness, and rubric quality, triaged P0/P1/P2.
05Delivered task catalog (public summary)
Delivered series we can disclose; full field descriptions on the datasets page:
| Series | Capability | Band | Scale | QA |
|---|---|---|---|---|
| HLE-Math | Academic reasoning · multimodal | L3 | 2,000 problems, scalable | Solver + cross-validation + double-blind |
| HLE-Physics | Academic reasoning · multimodal | L3 | 2,000 problems | Cross-validation + double-blind |
| HLE-Chemistry | Academic reasoning · multimodal | L3 | 1,000 problems | Cross-validation + double-blind |
| Frontier-Eng | Engineering optimization · code | L1 | Aligned to the original benchmark | Executable unit tests |
| Deep SWE / Frontier SWE | Repository-level engineering | L1 | 200 problems / week | Cross-validation + double-blind |
| ALE (Agents' Last Exam) | General agent | L2 | 200 problems / week | Unit tests + cross-validation |
| Automation Bench | Desktop automation · agent | L1 | 200 problems / week | Double-blind |
| Toolathlon | Tool orchestration · agent | L1 | 200 problems / week | Double-blind |
| Harvey LAB | Legal agent | L1 | 200 problems / week | Double-blind |
| Embodied suite (4 lines) | Grounding QA / SFT / annotation / DAgger | L2 | 30k–1M items | Cross-validation / double-blind |
06A question list for buyers
When evaluating any task-data vendor, ask — we hold ourselves to the same list:
- Are tasks originally authored? How do you prove they are outside pretraining distributions?
- Is difficulty quantified? On which model, under which pipeline?
- Does the verification environment ship with the tasks? Are results reproducible?
- Who runs QA? Is there a model-grading-itself loop?
- How broad is the contamination probe coverage?
07Work with us
DataMining was founded by PhD researchers from the School of Software at Peking University and has delivered task data to leading technology companies including ByteDance, Alibaba, and Tencent, across coding, complex reasoning, multimodal, and agent capability lines. If your team needs higher-discrimination training and evaluation tasks — or wants the measurement methodology and samples first — contact contact@datamining.cloud.
References
- Humanity's Last Exam (Center for AI Safety & Scale AI, 2025), arXiv:2501.14249
- SWE-bench: Can Language Models Resolve Real-World GitHub Issues? (2023), arXiv:2310.06770
- Introducing SWE-bench Verified (OpenAI, 2024)
- Deep SWE (Agentica Project)
- RL environments and reward hacking (Prime Intellect)
- Capability descriptions of Frontier-Eng, ALE-Bench (Sakana AI + AtCoder), APEX Agents, Program Bench, and Terminal-Bench follow their official releases.