Five Agents Generated Their Own RL Training Tasks, and Every One Failed
Letting agents write their own training data is the one step automated data synthesis most wants to close. An open benchmark, AutoDataBench, put five agents on the exam and none of them reached 0.2.
Recursive self-improvement (RSI) and data flywheels are the standard framing in the industry right now, and agent-generated training data is the key link in that story. The open benchmark AutoDataBench (GitHub repository StarDewXXX/AutoDataBench) recently ran a set of measurements: five agents produced reinforcement-learning training tasks across three domains, terminal-bench, tb-science and automationbench, on 24 source tasks. Under a 1-point scale, no agent's total score passed 0.2: kimi-k3 led at 0.184, then gpt-5.6-sol at 0.177, qwen3.8-max at 0.139, glm-5.3 at 0.130, and deepseek-v4-pro at 0.097. For buyers assembling RL task datasets in-house, the conclusion is hard to avoid: fully automated task production is not yet at a deliverable standard.
01What AutoDataBench measures
AutoDataBench measures one thing: whether the agent under test can replace a human data team and produce new training tasks for agent reinforcement learning. The unit of agent RL data is not a text pair, it is a task, and a complete task carries an executable environment, a verifier, and a difficulty matched to the target model.
The evaluation chain defines four independent roles: target model, agent under test, analyst, and reviewer.
- Target model: the object being trained, fixed at deepseek-v4-pro for this experiment.
- Agent under test: the data producer, which reads the source task and the target model's failure records, then builds a new task environment and verification script.
- Analyst: runs the front-end attribution, distilling a hidden rubric from the target model's six attempt trajectories and marking the decision points where it went wrong and how.
- Reviewer: once the agent under test delivers a new task, executes the verification and assigns the score.
The scoring formula is score = gate × difficulty × quality, and a zero in any term zeroes the total. The documentation is explicit that this is a proxy metric for the artifact and cannot prove in advance that a single task will help RL training; as an engineering acceptance standard for data, though, the design lands on the real problem.
gate: seven rules, any hit ends the run
Gate is the entry checkpoint and holds seven fatal problems: a verifier that does not check correctness, a leaked answer, an unsolvable task, a reskin, plagiarism, out-of-scope calls to an external model, and no valid task at all. Reskinning is the most frequent failure, meaning nouns are swapped and the substance is untouched. The test harness runs a text diff first to screen those out. Past experiments already showed that human review alone struggles to catch reskins with high text overlap, so automated front-end interception is necessary.
difficulty: measured, not annotated
Difficulty does not come from a subjective label. It comes from how often the target model solves the task across repeated runs, and the pass rate has to land inside a specified band. All correct means the model already has the capability and no gradient signal is left; all wrong means the task is beyond the current capability boundary and no positive trajectory can be sampled.
quality: evidence from the trajectory
Quality asks whether the new task hits the failure modes recorded in the rubric. The reviewer cannot judge from the prompt alone: it has to find evidence in the model's real execution trajectory that the task triggers the corresponding decision error before coverage counts as valid.
02The failure sits at both ends of the difficulty band
All five agents scored below 0.2 across the 24 source tasks. Decomposing the three multiplied terms locates the actual bottleneck in automated data production.
The attribution is clear: rubric coverage is close to saturated, and the failures concentrate at both ends of difficulty. One boundary on the sample: 24 source tasks, a single target model, a small scale, but the attribution structure is legible.
That says current mainstream models are not weak on the quality dimension. Given the target model's full failure trajectories from six attempts on the source task, the agent under test can identify the decision points where it fails and design an examination around that error. The real blocker comes after: turning that examination into an executable environment with a matching verification script, the agent cannot hold the task's actual difficulty steady. The new task, run against the target model, either passes every time or fails every time, and neither is usable as an RL training sample.
Recognizing a model's failure signatures is static text understanding. Controlling task difficulty is a dynamic systems engineering problem. In terminal operation, scientific computing and automation — the core agent RL settings — difficulty is not set by the prompt text alone: it comes from environment dependencies, tool feedback, state-space size and the tolerance boundary together.
Agents under test show two typical tendencies when building tasks. To keep their own verification script runnable, they simplify the environment and pre-load hints into the prompt, difficulty drops, the target model answers reliably, and difficulty scores zero. Push the other way and add file operations or complex dependencies, and latent problems appear: environment deadlocks, missing dependencies, missed boundary conditions; or the task chain grows past the target model's planning and context limits and it cannot finish at all, so difficulty fails the same way.
What the automated authoring agent lacks is a closed loop: calling the target model for real rollouts during generation, and adjusting environment resistance from the feedback. Without real trial and error to calibrate difficulty, output keeps collapsing toward the two extremes of all-pass and all-fail.
Field evidence: the tbs scientific computing task set. Tasks sticking at both ends of the difficulty band is not a quirk of an evaluation benchmark; it is what we handle every day. In a recent tbs scientific computing delivery, the target model DeepSeek-V4-Flash-Vision-Exp exhausted its budget on a single task across five consecutive attempts (test configuration: 80-step cap, 262,000 output tokens, 1,800 seconds, mixed temperature 0 and 0.6); on that one-sided estimate the 95% confidence upper bound on success rate is 45.1%; the trajectory viewer recorded 71 model turns on that task. The delivery kept three tasks out of eight candidates, preserved the evidence for every rejected task, and withdrew one task after an error was found in its reference answer. Pinning difficulty to the edge of budget exhaustion while keeping the task provably solvable depends on dense real test runs and elimination.
03Reading the scoring logic: what procurement actually accepts
AutoDataBench's gate, difficulty and quality line up with the acceptance logic we use when delivering RL task data.
Gate maps to the entry floor of a deliverable. Blocking reskins and plagiarism, blocking leaked answers, reproducible tasks, traceability across the whole chain: these are hard requirements for an industrial data package. Many buyers split acceptance into three tiers, and a single unmet Must-pass item voids the entire batch.
So a data package for a large buyer cannot be a pile of loose JSON files; it has to be an engineering artifact that passed full validation. Take our DeepSWE package: one package holds five tasks, 33 trajectory slots and 27 valid trajectories; the six invalid trajectories keep null fields and are not deleted; three empty patches are labeled rather than hidden, which keeps apply_failed=0; CRC checks cover every file, with a SHA256SUMS manifest and a PROVENANCE document recording the upstream dataset commit, the grader hash and the base commit of each repository. Invalid records stay visible, patches apply with certainty, provenance goes down to the hash. That is what gate checks look like when they land in a delivery process.
Difficulty maps to the window that only repeated measurement can pin down. Mature buyers do not accept synthetic data that has never been run against the target model. The procurement requirements for the Berkeley ALE benchmark (13 industry clusters, 55 subdomains, more than a thousand tasks) require at least one non-perfect score across pass@3; a later v2.0 task specification adds hard constraints: at least 50 solution steps, an SOTA machine score of 40 or below, and distracting material no more than one third of the package.
Holding a task inside that narrow band requires a data pipeline with large-scale compute scheduling and repeated real rollouts, with measured pass rates as the verdict. All of our deliveries use that measured standard: five DeepSWE tasks run 20 times with GPT-5.6-sol (xhigh), three successes; four FrontierSWE Go tasks, three successes in 16 attempts; two CHEMQ chemistry tasks at Pass@4=0. Only when the model version, attempt count and success count are all fixed can the resulting tasks drop straight into an RL training pipeline.
Quality is about hitting the model's real failure modes on purpose. Matching AutoDataBench's requirement that the reviewer extract evidence from the trajectory, a high-quality synthetic task has to aim at the target model's actual decision blind spots. When we build long-horizon tasks, we first collect the target model's full trajectory on a seed task, have domain experts locate concrete problems such as wrong tool calls, forgotten state or skipped reasoning steps, turn those into automatically checkable items, and design the environment obstacles backwards from there.
Even expert-authored tasks rarely land at the right difficulty on the first pass. Of the tbs scientific computing tasks above we kept three of eight candidates; most of the five rejected hit the same two-sided difficulty problem, with the target model either solving reliably or not at all. The mechanism is identical to what the agents under test ran into. The difference is only that our failures are filtered inside quality control, with rejection reasons and rollout records archived, and never reach the delivered package.
The bottleneck in automated data synthesis is not seeing where a model is weak. It is calibrating difficulty in a stable, controllable way. At the current stage, expert-authored tasks with strict quality control around real model rollouts remain necessary. That gap is exactly the routine work our industrial delivery handles.
If you are working on RSI, agent post-training or long-horizon agent evaluation and need matching task environments and data, write to contact@datamining.cloud.