When a Long-Horizon Trajectory Fails, Should You Delete It or Keep It?
Snapshot rollback lifts collection success from 0.0% to 18.7% at the hardest setting, but discards the branches that define the denominator. Why we keep the nulls.
One of our Deep SWE-style delivery packs contains five repository-level tasks with 33 historical runs, 27 valid. Six invalid runs remain in validation.json with their scores held at null. Three of the 27 finished with an empty patch; they are scored as zero on a clean baseline and flagged separately in metadata. The manifest reads PASS_WITH_LIMITATIONS, not PASS, and native_environment_verified: false is recorded in the same file. The README states plainly that this is a difficulty-screened development sample, not proof of superiority on an official benchmark.
The industry-standard shortcut would be a one-line filter before packaging: drop the six nulls, report 27, and ship a spotless 100%-valid bundle. We do the opposite.
01Rollback rescues the trajectory and loses the denominator
A November 2026 arXiv paper on CUA-Master (2611.03126) proposes recursive improvement for computer-use agents, producing 127K training samples and a 27B model. Its snapshot-rollback mechanism estimates the earliest point of deviation from verifier residuals, the sampled trajectory and the executable environment, restores the last legal snapshot, keeps the stable prefix, discards the bad suffix, and retries with feedback that does not leak a privileged solution. After the branch search, only the single most efficient successful path is retained.
For training, that trade-off is sound. Failure usually comes from a local misstep, so rerunning an entire trajectory wastes compute; successful paths also contain redundant actions. The collection numbers are striking. Under the same retry budget, success rates for restart-from-scratch versus rollback-with-feedback move from 74.0% to 95.4% at (5,1), 31.5% to 83.7% at (20,1), 19.3% to 56.9% at (20,3), 1.9% to 32.8% at (20,5), and 0.0% to 18.7% at (50,8), where K is the number of capability primitives and N the number of applications.
But if you are buying the data, the discarded branches matter. To rescue one positive example you also throw away every branch that died in the same search. You see a numerator of one and no denominator: how many rollbacks happened, at which step, and how many attempts it took. Deliver only the numerator and every task looks like a high pass rate.
02Failed runs are the raw material for difficulty calibration
We keep failures for three concrete reasons. First, pass@k is meaningless without the attempt count it came from. Second, the distribution of failure steps tells you whether a task is hard because of reasoning, tool use or environment noise. Third, retired tasks and withdrawn reference answers are the audit trail a buyer needs when a score looks too good.
That is why our packs carry SHA256SUMS, PROVENANCE.md, package-level CRC and byte checks, with invalid runs left in place and empty patches labelled rather than deleted.
03In a desktop environment, the initial state is part of the task
A GUI agent's result depends on the exact starting snapshot: window layout, cached credentials, file system state, installed versions. If the initial state is not pinned and hashed, a "successful" trajectory cannot be reproduced, and a failure cannot be attributed to the model. Snapshot discipline is not housekeeping; it is the definition of the problem instance.
04Evidence costs money, so keep what is attributable
Full-fidelity trace capture at scale is expensive. The rule we use is to retain evidence that supports an attribution: verifier residuals, the deviation step, the environment hash, and the failure class. Raw video of every idle frame does not make a dataset more verifiable, it just makes it larger.
05How to audit a vendor's failure records
Ask for the nulls, not the highlight reel. Ask which runs were withdrawn and why, whether the reference answer was ever revised after a model exposed a flaw, and whether pass rates are reported with their attempt budget. Ask for the manifest status verbatim — PASS_WITH_LIMITATIONS is a stronger signal of maturity than an unqualified PASS. And check that the grader's own error rate is measured, because a verifier that misfires quietly turns both successes and failures into noise.
About DataMining Founded by PhD researchers from the Institute of Software, School of Computer Science, Peking University, DataMining builds original task data and executable evaluation environments for frontier models and agents, with reproducible rubric specifications and bootable Harbor Task verifiers. Benchmarks named here are public third-party benchmarks; we align capability requirements without copying tasks. Visit datamining.cloud or write to contact@datamining.cloud.