Post-Training in Deep Water (II): Catastrophic Forgetting and the Fully Automated Data Alchemy Pipeline

DataMining · November 9, 2026

In the previous article we examined the core tension in post-training: in real business settings without ground-truth answers, how RLVR (Reinforcement Learning with Verifiable Rewards) and Rubrics (atomic scoring criteria) convert vague human expertise into dense reward signals a model can learn from.

In production, however, teams quickly hit a harder wall: even with perfect Rubrics, a model still drifts or simply forgets during training.

Once the algorithmic gains are exhausted, the second half of post-training becomes a systems-engineering problem of data mixture, training order, and environment construction. Drawing on recent discussions on X (formerly Twitter) and first-hand industry observations, this article unpacks the overlooked technical hazards in post-training pipelines and the paths major labs are taking around them.

0101 The Frequent-SFT Trap: Why Feeding Only Successful Trajectories Leads to Catastrophic Forgetting

When building an agent or a domain-specific model, the intuitive approach is to collect a large set of high-quality expert success trajectories and keep applying supervised fine-tuning (SFT).

But Charlie O'Neill, who leads Baseten Base Labs, recently offered a counterintuitive observation: if you repeatedly SFT on small batches of successful trajectories only, whether on-policy or off-policy, the model shows clear catastrophic forgetting after a few hundred updates. It loses general capabilities learned during pre-training, and even its language fluency degrades.

Under the hood: SFT is maximum likelihood estimation. It forces the model to fit every individual token in the teacher trajectory. The model therefore absorbs not just the correct logic but also the teacher's phrasing habits, redundant transitions, and even small distribution shifts. As this fine-tuning repeats, the original parameter space is overwritten and general knowledge is washed out.

By contrast, RL has a distinct advantage in preserving general capability while adjusting policy. RL does not force the model to imitate every token; through the policy gradient it nudges the probability distribution only on the correctness of the final outcome. It changes the parameter space very little, yet moves the model precisely in the right direction.

Engineering takeaway: SFT is better suited to injecting new knowledge or formatting outputs (for example, teaching a specific JSON schema); RL is better suited to reweighting policy and improving complex reasoning. The two are not interchangeable and must alternate under a carefully designed curriculum.

0202 The Gap Between Benchmark Distributions and Real Environments: How to Build a Reverse-Solved Testbed

OpenAI co-founder John Schulman once pointed out a fatal weakness in today's post-training environments: we build far too many hard environments and far too few real ones.

Many benchmarks are carefully designed puzzles: the task is extremely hard but verification is trivial (a complex olympiad problem has a single answer). Real work, such as multi-round code review or cross-department compliance checks, is full of ambiguity, conflicting objectives, and dynamic feedback. If RL runs only on the benchmark distribution, a model may match its teacher on the leaderboard yet behave like a book-smart test-taker in production.

How do we build training environments that are both genuinely hard and faithful to the real world? Charlie proposes an illuminating idea: reverse environment construction.

• Forward solving: give the model a well-defined problem and let it derive the answer.
• Reverse construction: first define a complex data-generating process with hidden latent variables, hide those variables from the model, then require it to infer the underlying mechanism through observation, experimentation, and reasoning.

An example: a human engineer spends three days locating and fixing a deeply hidden concurrency bug in a large legacy codebase. A lab can package that repair process as an environment, giving the model only error logs and partial code snippets. The human who set the task knows the root cause (the latent variable), but the model must spend a large number of tokens on multiple rounds of code search, adding logging, and even writing test cases before it can work backward to the answer.

This information asymmetry makes it possible to construct environments that are both extremely hard and well structured. The model cannot guess the answer from statistical patterns; it has to do real reasoning work.

0303 The Deep End of Data Governance: Safety Guardrails Can Be Trained Away

Post-training data has moved beyond purely technical concerns into security and data sovereignty. Two recent debates on X expose the double-edged nature of RL's power.

1. An offer to fund malicious RL post-training: security researcher @jun_song disclosed that an anonymous black-hat team offered an astronomical sum for exploit data, intending to run malicious RL post-training on open-weight models.

1. Implicit appropriation of private data: OpenAI's updated Data Controls terms sparked controversy, noting that even when users turn off the option to improve the model, content from certain advanced sessions (Trusted Access) may still be used for training.

Under the hood: RL works by amplifying the weight of high-reward trajectories. If maliciously crafted jailbreak or exploit trajectories are assigned high reward, a model's safety alignment can collapse completely within a few epochs. It quickly learns to route around existing guardrails, because in that particular RL environment, producing malicious code earns a higher score.

Countermeasure: to resist single-dimension preference overfitting, researchers are exploring PlurPO (Pluralistic Preference Optimization). The idea is to simulate multiple stakeholders when building the preference dataset, such as safety reviewers, domain experts, and ordinary users. If any party vetoes a response, that trajectory is neither generated nor used for training. Recent reports show this cuts social sycophancy or safety-compromise behavior on specific metrics by an average of 89%.

0404 The Endgame: From Hand-Written Rubrics to Automated Post-Training Loops (RSI)

Look further ahead and the limits of hand-writing dozens of Rubrics and hand-building environments become clear. The endgame of post-training is necessarily an automated closed loop of Recursive Self-Improvement (RSI).

Two recent frontier signals sketch what this looks like:

1. A fully automated post-training loop: Applied Compute recently demonstrated its Ari Autoresearch system, claiming an AI agent can run a fully automated post-training loop for 16 hours straight with no researcher intervention.

1. World-model-driven data replenishment: RoboCoach, from teams at Renmin University of China and Tsinghua University, uses a world model to diagnose a robot's current weaknesses. When the model repeatedly fails a subtask, the system automatically turns that failure trajectory into a targeted demonstration request and adds it to the training pool. After 3 rounds of targeted replenishment, task success rose from 13.3% to 75.0%.

The automated post-training pipeline of the future looks like this: the model executes a task in a real environment → it fails or scores low → the system automatically analyzes the failure and dynamically generates targeted Rubrics and counterfactual demonstrations → this new data is injected seamlessly into the next round of GRPO reinforcement learning → capability improves incrementally → the model goes back into the environment for testing.

In this loop, the human expert's role shifts from hand-writing every scoring rule to designing the evaluation framework and its constraints. Models learn to recover from their own mistakes instead of mechanically copying successful trajectories, as many pipelines do today.

0505 Conclusion: Post-Training Is a Discipline of Cybernetics

Looking back at the evolution of post-training, a clear curve emerges: from blindly piling up SFT data, to DPO/PPO preference alignment; from impressionistic human scores, to RLVR and atomic Rubrics; and now to guarding against catastrophic forgetting and exploring fully automated, dynamic data replenishment loops.

The second half of post-training is no longer a few researchers tuning hyperparameters in a lab. It demands three things from a team at once:

• A data scientist's instinct: precise control over the mixture weight (λ) of pre-training, mid-training, and RL data.
• A domain expert's rigor: translating tacit business logic into machine-executable, positively and negatively balanced atomic Rubrics.
• A security engineer's vigilance: pluralistic preference checks and red-teaming at every data ingestion point, to keep the model's distribution from being poisoned.

In the world of reinforcement learning, whatever distribution you feed the model is the reality it gives back. Whoever first builds a data alchemy pipeline capable of self-diagnosis and self-repair will forge genuinely irreplaceable AI agents on the real commercial battlefield.

About DataMining A team of PhD researchers from the Software Institute, School of Computer Science, Peking University, focused on high-quality original task data and executable evaluation environments for LLM and agent R&D. We deliver not just data, but industrial-grade Rubric and Harbor Task verification systems. ▸ Long-horizon code and engineering simulation: Deep SWE, Frontier SWE, Frontier-Eng, ALE / Terminal-Bench, with stable delivery of 200 tasks per week within 7 working days. ▸ Extreme mathematics and professional agents: a strict multimodal HLE production line (5,000+ items in stock); Harvey LAB, Automation Bench, Toolathlon, and 35+ atomic-Rubric industry workflow packs. ▸ System infrastructure and embodied real robots: standardized Windows 11 / Ubuntu desktop-agent VM images, plus million-scale embodied Grounding / SFT / DAgger real-robot custom production lines and cyber data. The research, datasets, and benchmarks cited here are all public third-party work; we align only with their capability requirements and difficulty ranges, and do not copy their items. Capability descriptions reflect our own internal conclusions under the same methodology and do not represent official third-party findings. 🌐 Website: datamining.cloud｜📮 Sample requests and technical contact: contact@datamining.cloud

📚 Figure Sources

[1] Retaining by Doing: The Role of On-Policy Data in Mitigating Forgetting, arXiv:2510.18874v3, Figure 1 / Figure 2

[2] Mitigating Social Sycophancy via Pluralistic Preference Optimization (PlurPO), arXiv:2610.02568v2, Figure 1

[3] RoboCoach: World Models as Active Coaches for Compositional Robot Skills, arXiv:2609.39685v1, Figure 1 / Figure 3