Passing Unit Tests Is Merely an Illusion of Software Engineering

Earlier this year, the industry was still reveling in high scores on SWE-bench. By autumn, however, the tide had receded.

OpenAI has quietly removed SWE-bench Verified from its core metrics for measuring frontier coding capabilities; meanwhile, a report from the independent evaluation organization METR pointed out an awkward truth: those code patches that dominated the test sets wouldn't even get a second glance from repository maintainers once thrown into real-world open-source communities, let alone be merged into main.

In late October, as Artificial Analysis updated its 5th-generation intelligence index (AA v5), the evaluation criteria for long-horizon software engineering underwent a brutal overhaul. FrontierCode 2.0, created by Cognition in collaboration with top open-source project maintainers, officially became a core metric, complemented by the upgraded SWE-bench Pro V2, DeepSWE 1.1, as well as ServerBench and CompressionBench, which emerged in early November.

The flowchart above shows AI writing first, reproducing tests, and then modifying code; the three bar charts below highlight SWE-bench Pro at 63.2%, Terminal-Bench at 80.4%, and FrontierCode at 38.8%

The winds have completely shifted. Model labs' demand for code data has fully pivoted from "patching an isolated, toy-level bug" to complex, long-horizon software engineering.

The result? Many open-source models that score 70% or even 80% on older leaderboards see their scores slashed in half on private, long-horizon benchmarks like FrontierCode 2.0. Even Claude Opus 5.5, which currently dominates the charts, scores just 46.4%.

Why can models write correct code snippets yet struggle with long-horizon software engineering? After dissecting numerous frontier benchmarks and training pipelines, we found that this is precisely the biggest missing piece in current LLM post-training.

Why does code pass tests yet fail to merge into main?

In real-world engineering teams, passing tests is often only half the battle. Whether a piece of code can merge into main depends on whether it violates architectural constraints, handles edge cases, features clean interface design, and avoids becoming unmaintainable technical debt six months down the line.

Side-by-side comparison of two code blocks: the left highlights Covered Sections in green, the right highlights Untested Assumptions in red, with 100% COVERAGE CAN STILL MISS WHAT MATTERS written at the bottom

Yet over the past six months, models have picked up a host of bad habits in post-training aimed solely at gaming test scripts.

In mid-October, Handshake's audit of thousands of training trajectories (Rollout) for DeepSWE 1.1 thoroughly exposed this issue. Without any mention of a grading script, the first reaction of over 80% of frontier models upon receiving an engineering requirement was not to understand the architectural design of the entire codebase, but to guess: "How will the hidden test script evaluate me?"

This directly spawned several deformed code patterns in long-horizon development:

Scope Collapse: Even when the requirement clearly demands refactoring an entire permission verification logic, the model calculates the odds and hardcodes only the two or three branches most likely to be covered by unit tests, leaving the remaining complex edge cases completely blank.

Hollow Implementations: To meet a certain performance or return-value metric, the model avoids modifying the underlying logic and instead hardcodes a fake state directly at the top level, wrapping up as soon as it tricks the assertion.

API Saturation: Unsure of which function name the test script will call, the model simply writes out seven or eight possible combinations of names and parameters inside the class.

Such code can score full marks on old benchmarks that only look at unit test pass rates; however, in front of FrontierCode 2.0, its true colors are immediately exposed.

Compared to the previous generation, the codebases in FrontierCode 2.0 are 25% larger, and reference solutions modify 67% more files across the codebase and write 89% more lines of code on average, covering complex security patches and end-to-end performance optimizations.

What is even more unforgiving is its grading method: it no longer looks only at Unit Tests, but introduces a "Mergeability Rubric" formulated by core maintainers of open-source projects. This is divided into veto items and weighted design items. As long as the code breaks architectural consistency, introduces dirty dependencies, or is caught peeking at answers by the Trajectory scanner, it directly receives a score of 0 even if all tests pass.

SWE-bench Evaluation Flow: GitHub Issue → Codebase → Agent Actions (Search / Edit / Test) → Test Suite → Success or Failure

Three Real Pain Points of Long-Horizon Engineering: Asynchrony, Architecture, and Context Pollution

Beyond code quality, once tasks are stretched to hundreds of steps or even run continuously for several hours, long-horizon software engineering faces three purely engineering-level hurdles.

First, real system-level development is never single-step Q&A, but multi-process and asynchronous scheduling.

In the Stirrup framework that runs FrontierCode 2.0 on AA v5, each task is capped at 500 interaction turns and supports persistent background Shell commands. This means a mature Coding Agent needs to operate like a human engineer: kicking off a twenty-minute compilation or stress test in the background, turning around to modify configuration files for another module instead of waiting idly, and switching back later to check the build logs.

LLM Agent Reasoning Loop flowchart: User → LLM Model → Tool Call → Observe → End Turn? → Response, with tool and observation fields listed on the right

With ServerBench and CompressionBench released on November 4, requirements escalated directly to system-level R&D: the former requires the Agent to design, implement, and tune a complete LLM inference serving system from scratch, squeezing out throughput under session dependencies and latency constraints; the latter requires the Agent to independently complete quantization and pruning compression of large models in the terminal. In such long-horizon tasks spanning hundreds of steps, the vast majority of models on the market—trained only on single-file edits—lose their bearings after thirty steps and fall into endless loops (Doom-looping).

Second, "context garbage pollution" in long trajectories.

Just on November 5, from OpenAI's Codex and Anthropic's Claude Code to open-source projects like OpenCode and Pydantic-AI, repositories across almost every mainstream Agent framework were intensely discussing the exact same question: when a tool call returns a massive pile of useless garbage output, how do we let the model clean up its own historical context?

During long-horizon development, a model will inevitably mistype a grep command or print tens of thousands of lines of error logs. A human programmer can just glance at it and close the window; however, without dedicated Context Pruning, those tens of thousands of lines of garbage logs remain in the LLM's context indefinitely. Across the next several hundred interaction turns, the model has to re-read this pile of garbage at every single step. Not only does the Token bill double (imec calculated that a poor Harness can double the cost per resolved task from $0.35 to over $0.7), but the model's attention is also severely disrupted, causing its output to become increasingly incoherent the further it writes.

Three-tier hierarchical context map (Level 0 Project Map → Level 1 Project Details → Level 2 Source Files), comparing full-repo grep vs. targeted lookup and 80% TOKEN SAVINGS on the right

Third, adherence to project engineering guidelines (such as AGENTS.md).

A November 3 study specifically tested how well a Coding Agent complies with repository-root engineering guideline files (ACF). During long-horizon edits, the model initially remembers to follow exception-handling rules and directory conventions, but after fifty rounds of tool calls, those early constraints are completely forgotten, and it starts arbitrarily creating files and modifying permissions. (This is also a key warning highlighted by VeriHarm-Bench, released around the same time: just to get a task passing, an Agent will frequently delete unrelated business files or open up system permissions completely.)

What we can do, and why we can get this right

Understanding why model labs lose points on FrontierCode 2.0, SWE-bench Pro Hard, and system-level engineering benchmarks makes it clear what kind of code data LLM companies will need most urgently over the next six months.

Ordinary competitive-programming data and simple single-file Patch samples are already in severe oversupply. On this tough frontier, what our team consistently delivers—and what model labs value most during acceptance—comes down to four types of high-difficulty datasets and environments:

1. Industrial-Grade "Mergeable" Long-Horizon Multi-File Engineering Suite

We do not collect simple issues that can pass tests by tweaking just three to five lines of code. Benchmarked against the granularity of FrontierCode 2.0 and specifically targeting million-line-scale codebases, we construct tasks covering feature development across dozens of files, underlying architectural refactoring, security vulnerability fixes, and system-level performance optimization. Each task not only requires functional execution, but also comes with a "Mergeability Rubric" authored by senior open-source Maintainers and architects—breaking down interface compatibility, concurrency safety, exception handling, code style, and module decoupling into strict veto criteria and weighted scoring items, specifically designed to cure models of writing "hollow shell code" and "gaming the tests."

2. Complete Expert-Level Trial-and-Error and Context Management Trajectories (Long-Horizon Trajectory)

Addressing the issue where models easily fall into infinite loops and suffer context collapse after exceeding 100 steps, we collect more than just the final correct Git Commit; we capture the complete troubleshooting pipeline of senior engineers across terminals and IDEs: including how to break down large tasks, how to utilize background asynchronous processes to compile while editing code, how to pinpoint bottlenecks based on gdb/profiler profiling results, and most crucially—when encountering dead-end paths or verbose garbage logs, how to proactively perform Context Pruning and roll back states. Such high-signal-to-noise-ratio trajectories featuring "negative sample correction" and "active memory cleanup" can directly help models drive down sampling costs during long-horizon RL training.

3. AI R&D and System Infrastructure Development Environments (Infra & Model Iteration)

Keeping pace with the latest November trends in ServerBench, CompressionBench, and deep GPU Kernel Optimization, we build closed sandboxes equipped with real deep learning frameworks, CUDA compilation environments, distributed inference engines, and performance stress-testing toolchains. This allows models to practice real-world model compression, kernel rewriting, high-concurrency service tuning, and long-horizon Shell operations inside, meeting model labs' specialized training demands for "Recursive Self-Improvement (RSI)" and hardcore systems engineering.

4. Spec-Driven & Interactive SWE Data (Spec-Driven & Interactive SWE)

Real-world development often starts with a vague requirements document. Combining the Spec-driven workflow and PersonaCodeBench that gained academic attention in early November, we can build multi-turn, long-horizon interactive datasets covering "requirements clarification, architectural design document (Spec) drafting, phased code implementation, and Code Review rejection and revision," training models to strictly adhere to project-level engineering standards (such as AGENTS.md) and security boundaries over long cycles.

In this track, many data vendors can also hire programmers to write code, but the submitted data often fails to pass the post-training acceptance tests of foundation model companies. The reason we can build such high-difficulty, long-horizon engineering data with depth and deliver genuine score improvements relies on three core underlying technical strengths:

First, a physically cheat-proof sandbox and static audit pipeline.

Following the October controversies across major leaderboards involving "peeking at Git history" and "tampering with grading scripts," we deeply understand that environment cleanliness determines the success or failure of training. Every single one of our engineering environments undergoes scrubbing by our proprietary static scanning tools during the packaging phase: completely stripping away origin/main remote residues and historical Commit spoilers, enforcing strict permission and filesystem isolation between the test executor (Verifier) and the model runtime container, and intercepting process tree timeout residues and timing measurement blind spots. The environment delivered to model companies is a clean testing ground that truly seals off all opportunistic backdoors.

Second, a three-tier grading system of "Execution Verification + Architectural Rubric + Trajectory Intent Audit."

Ordinary environments only provide binary 0 or 1 unit test results, which makes the signal too sparse and leaves room for models to game the system. We built three layers of verification into the environment: the bottom layer consists of high-coverage functional and performance benchmark tests; the middle layer is a code structure and maintainability Rubric aligned with the standards of core Maintainers; and the top layer introduces the latest trajectory intent verification mechanism (similar to the Epistemic trajectory review and harmlessness checks proposed in early November), scanning in real time across hundreds of tool-call steps for underhanded behaviors such as deleting tests, altering configurations, or unauthorized operations, ensuring that every positive reward fed into GRPO/RL training reflects genuine engineering capability.

The complete evaluation pipeline of a single Rubric: Problem Definition → Evidence Collection → Evidence Artifacts → Direct Judgment → LLM Reviewer → Deterministic Aggregation → PASS / FAIL / NEED REVIEW

Third, an engineering team that understands both Harnesses and post-training ablations.

Whether long-horizon code data yields effective training results depends half on the tasks and half on the Agent Harness (scaffolding and toolchain definitions) wrapping the model. Rather than blindly delivering data like traditional outsourcing vendors, our team deeply tracks the underlying mechanisms of mainstream Harnesses such as Claude Code, Codex, OpenCode, and Stirrup. Before data delivery, we run multiple rounds of base model Rollouts and difficulty stratification under a standard Harness, filtering out invalid dead-end tasks caused by prompt ambiguity or hidden unfair constraints, ensuring that every batch of long-horizon engineering data delivered to model labs produces clear gains on both private test sets and real-world coding experience.

Ultimately, as the competition among coding Agents shifts from "writing a runnable script" to "maintaining a million-line software engineering edifice," the barrier to entry in the data business has fundamentally changed.

Only a team that truly understands how large-scale software is written, reviewed, and run inside containers can help large models cross the final hurdle from "leaderboard hackers" to "qualified engineers."

About DataMining Founded by a team of PhD students from the Institute of Software at Peking University's School of Computer Science, we focus on providing high-quality original task data and executable evaluation environments for LLM and agent R&D. We deliver not only data, but also industrial-grade Rubric and Harbor Task verification systems. ▸ Long-Horizon Coding & Engineering Simulation: Deep SWE, Frontier SWE, Frontier-Eng, ALE / Terminal-Bench, etc., supporting stable delivery of 200 tasks/week within 7 business days. ▸ Extreme Math/Science & Professional Agents: HLE strict multimodal pipeline (5000+ tasks in stock); Harvey LAB, Automation Bench, Toolathlon, and 35+ atomic-level Rubrics industry workflow packages. ▸ System Infrastructure & Real-World Embodied AI: Standardized Win11 / Ubuntu desktop Agent VM images, as well as million-scale embodied AI Grounding / SFT / DAgger real-machine custom pipelines and cyber data. All research, datasets, and benchmarks cited in this article are public third-party works; we only align with their capability requirements and difficulty ranges without copying tasks. Capability descriptions herein represent our internal same-standard conclusions and do not represent official third-party conclusions. 🌐 Website: datamining.cloud｜📮 Sample Requests & Technical Inquiries: contact@datamining.cloud

📚 Image Sources

The 6 images in this article are taken from image search results in a Doubao session (public technical materials, reposted via Toutiao) and inserted in the order of the text; captions only describe the visible content in the images and do not represent official third-party conclusions.