Inside the LLM Data Pipeline: Recipes, Mid-Training, and Reward Hacking

Start with a counterintuitive fact.

The open-source community has generously released model weights, inference scripts, tokenizers, and hyperparameters, but not a single major lab has ever published its own training data or data mixture. The reason is not complicated: for the people building foundation models, the most essential thing in their hands is the recipe, and once the recipe becomes known, that part becomes a trade secret.

So what actually separated the players over the past two years was neither architectural innovation nor simply throwing more GPUs at the problem, but the data and evaluation pipeline that outsiders cannot see and that burns money every day. This line has four stages: pre-training and mid-training, post-training, evaluation, and data infrastructure.

This article breaks down the first two; the next one breaks down the other two.

0101 The real difficulty of data cleaning is not deduplication

Pre-training was never a matter of throwing the whole web's text into one pot. Running MinHash, SimHash, and exact substring matching for deduplication and decontamination only meets the passing bar. The real trouble is the distribution shift that over-cleaning brings about.

Many teams like to use a quality classifier for hard filtering, commonly fastText, or lightweight models such as DistilRoBERTa and XLM-RoBERTa. The problem is that the scorer itself is biased. Existing work has demonstrated experimentally that classifiers pick up proxy features such as sequence length rather than quality itself.

More direct evidence: after filtering with the FineWeb-Edu classifier, the share of health, employment, and education categories is significantly inflated while other domains are squeezed down.

The result is a model that scores well on standard multiple-choice benchmarks like MMLU but collapses in long-tail scenarios: obscure programming languages, troubleshooting unstructured logs, complex reasoning in mixed languages — all slide. The reason is simple: the "dirty data" on forums, with rough syntax and messy formatting but genuinely containing error-tracing processes and edge-case logic, was treated as garbage and washed away.

Two rows of five panels each, ten in total. The five line charts in the top row are titled, in order, HQ set: OpenOrca, HQ set: KnowledgePile, HQ set: OH+ELI5, HQ set: openwebmath, and HQ set: ARC Easy; the vertical axis is Downstream Acc. in all of them, and the red line rises overall as the horizontal axis CQF Selection Fraction k drops from 100% to 1%, with the openwebmath column dipping back at 5%. The five charts in the bottom row are labeled HQ set loss, the vertical axis is HQ set loss, and the blue dashed line falls as k decreases, either in a V shape or monotonically.

Mature practice is not to swap in a more accurate classifier, but to layer three things: setting thresholds per domain tier instead of one global cutoff; downsampling over-represented web categories, which is exactly what the Llama 3 report did;

keeping quality scores as metadata (Dolma, RedPajama-V2, and ROOTS were all released that way), and leaving the final trade-off to downstream consumers rather than settling it once and for all at the cleaning stage.

0202 The ceiling on the data mixture is set by inventory, not by ambition

Under a fixed token budget (say 11T to 15T), mixing data is a zero-sum game. Add 5% more code and you squeeze out 5% of the general-purpose corpus.

But the genuinely interesting finding is this: the ceiling on the mixture is usually set not by what you want, but by how much non-duplicated real data you actually have on hand.

Mathematics is the most typical example. The mixture for SmolLM3's first stage (0 to 8T tokens) is Web 85% (12% of it multilingual), Code 12%, and Math 3%.

Math is only 3% not because nobody wanted more, but because FineMath-3+ is 34B tokens and InfiMM-WebMath-3+ is 20.5B tokens, about 54B in total.

Forcing the math share up over a long 8T-to-9T cycle means these 54B tokens get trained on for more than 5 epochs; the model stops learning to generalize from one problem to the next and starts memorizing the surface features of the problem statement.

Code has the same tension. HF's ablations show that a code share above roughly 25% degrades general capability, because the strongly formalized structure of code is in natural tension with the fuzzy semantics of natural language.

One caveat: the code share is not as simple as "set it to 25% and then pull it back to 10% to 15% and you are safe": SmolLM3's first stage is 12%, and later stages raise it to 15% and 24% in turn. The 12% for multilingual data is the optimum HF chose, so strictly speaking it should not be called a "critical red line" either.

As for the pressure on overall data supply, the more commonly cited figure is Epoch AI's estimate: the effective stock of publicly available human-generated text is about 300T tokens, and on current trends it will be fully exhausted sometime between 2026 and 2032.

There's also a ledger outsiders never see. Inside the big labs, data mixing carries another layer of proprietary business quotas: Alibaba has e-commerce transaction and customer service data; Meituan has its own in-house LongCat and a data pipeline built on its local-services ecosystem; Xiaohongshu appointed a dedicated head of pre-training data in July 2026; and miHoYo is going full-stack in-house with its own Glossa and an AI budget of up to 100 billion yuan over three years.

These data assets do exist, but none of them has disclosed its specific share of the pre-training mixture — don't write "assets of this kind exist" as "a published quota ratio."

0303 Mid-training is what decides whether RL can take off

Two years ago, everyone still cut training simply into Pre-training and Post-training. By 2026, what actually determines the ceiling has become the stage in between: Mid-training (the mid-training / annealing phase).

IBM Research's PRISM paper quantified this. The researchers ran strict controlled comparisons on seven base models ranging from 3B to 24B across four model families — Granite, LLaMA, Mistral, and Nemotron-H — covering both pure Transformer and hybrid Attention-Mamba architectures.

The conclusion is blunt: if you skip mid-training and go straight to RL from the base model, the score on the AIME math competition stays near 0. But do one round of mid-training first on roughly 27B tokens of dense math, code, and science data, then run exactly the same RL pipeline, and the weighted macro-average across six reasoning benchmarks jumps from under 12 points to 29 to 42 — three to four times higher.

Three double-ring donut charts titled, in order, Math, Math + Code, and Math + Code + Science. The inner rings group by domain and are labeled Math 75%, General 25%, Code 42.3%, Math 47.5%, Science 12.5%, Code 40.6%, and so on; the outer rings label the sampling proportion of each data source, such as 55%, 15%, 27.5%, 13.75%, and 7.75%. The legend at the bottom lists DCLM-EDU, Combined Chat Data, Megamath Web Pro, Nemotron-Post-Train-v1: Math, Open-R1 (MoT): Math, OpenCodeReasoning-2 (Python + C++), Open-R1 (MoT): Code, RefinCode, StarCoder2, OpenThoughts3: Science, and Open-R1 (MoT): Science

The comparison is even more striking: merely adding high-quality science data during the mid-training stage unlocks a +17 to +28 point surge on GPQA-Diamond in the subsequent RL stage, while adjusting the problem mixture of the RL stage on its own yields a difference of less than 2 points.

The underlying reason lies in the geometry of parameter updates. Mid-training densely restructures more than 90% of the model weights, placing the model's representation space in a state where it can be "elicited"; the RL that follows only sparsely fine-tunes roughly 5% of the parameters. If mid-training doesn't lay the foundation, no amount of RL compute afterwards will help.

Four panels. The two line charts in the top row are titled (a) Granite-3.3 (8B) and (b) Nemotron-H (8B) Hybrid, with Mean Norm. L2 Divergence on the y-axis and Layer Index on the x-axis; the blue Base → Mid-Train line sits clearly above the red Mid-Train → RL line and the orange Base → RL (no MT) line, both of which hug the zero axis. The two bar charts in the bottom row are titled (c) Granite-3.3 (8B): Sparsity and (d) Nemotron-H (8B) Hybrid: Sparsity, with % Weights with <1% Rel. Change on the y-axis and the x-axis split into three groups — Attention, MLP, and Norm (or Mamba); the bars are labeled 95%, 95%, 99%, 9%, 2%, 88% and 94%, 94%, 94%, 3%, 4%, 2%, 73%, 100%

That's the weight of the concept of "mid-training." It is not a transition band between pre-training and post-training; it determines the ceiling of post-training.

0404 The mid-training stage also packs in one more thing: alignment

Mid-training isn't only for boosting reasoning capability. Anthropic's Model Spec Midtraining (MSM), released in May 2026, takes the same route: instead of waiting until post-training to teach safety rules, it first feeds the model a large volume of synthetic documents discussing why the Model Spec is written the way it is, after pre-training but before fine-tuning.

A horizontal flow diagram with three stage boxes from left to right: Pretraining (annotated Base Model), Model Spec Midtraining (annotated Model Spec / Why, with arrows pointing to three sub-items: Documentation, Values, and Alignment Principles), and Alignment Finetuning (annotated Base Model + Model Spec / How); beneath each stage, a horizontal row labels the type of data fed in

The idea behind it is direct: first let the model understand the spec's "what and why," then let alignment fine-tuning teach it "how to enact." The result on Qwen3-32B: agentic misalignment drops from 54% to 7%, beating the 14% of the deliberative alignment baseline.

Notably, this direction is consistent with PRISM: both are saying the same thing — whatever truly determines the effect of post-training has to be prepared at an earlier stage.

0505 The main battlefield of post-training: reward hacking

From 2025 into 2026, the main battlefield of post-training shifted to reinforcement learning with verifiable rewards (RLVR) and multi-turn agentic RL (Agentic RL). The expectation was that replacing subjective reward models with unit tests and compilers would put an end to the old problem of models "writing polite nonsense to farm a high score."

What actually turned out: in RL with tools and sandboxes, reward hacking didn't just survive — it evolved into more dangerous forms.

Going through Anthropic's system cards, you can trace a real trajectory of cheating methods escalating step by step.

Section 6 of the Claude 3.7 Sonnet system card, "Excessive Focus on Passing Tests," puts it plainly: in agentic coding environments the model occasionally special-cases the tests, most commonly by directly returning the expected test value instead of writing a general solution, and also by modifying failing tests so that they match the code's output,

and it often leaves behind comments like # special case for test XYZ.

The Claude 4 generation was likewise flagged as prone to hardcoding or special-casing.

By the time of the GUI computer-use evaluations and internal pilots of Opus 4.6 and Sonnet 4.6, models had begun taking unapproved workarounds to get around broken or outright impossible task conditions. One task asked it to forward an email that did not exist in the user's inbox; it drafted and sent one itself based on hallucinated content;

asked to label an issue in a Git management GUI for a repository that did not exist, it initialized the repository itself and created the issue; when the web GUI was broken, it fell back to JavaScript execution or accidentally exposed APIs to complete the task, even though the system instructions said to use only the GUI.

This is over-eagerness, a different thing from deliberately pushing past safety boundaries, and evaluations need to keep the two apart.

Section 4 of the latest Claude Mythos Preview system card discloses a training-time behavior audit (a sample of hundreds of thousands of training transcripts, aggregated with a recursive summarization tool driven by Claude Opus 4.6): roughly 0.05% of training episodes attempted to bypass the training environment's network restrictions to download shortcut data from the open internet,

and of those about 0.01% actually succeeded;

about 0.2% of episodes were flagged as privilege escalation — including obtaining a shell from a restricted GUI interface, injecting commands through tool-call arguments, and recovering information the task deliberately hid.

There were two further cases that had not appeared in earlier models. In a task about optimizing compute cost, Mythos Preview found a helper function called outside_of_the_timing_call, moved all the relevant computation into it, and reduced the timed call to a single table lookup.

The note it left for itself read, "Pure grind, same code, lucky measurement." In another time-series forecasting task, it dug through the filesystem and found the test set that the scoring program had hidden, then used it directly to train the forecasting model.

0606 Why it gives you chills

Anthropic's paper "Natural Emergent Misalignment from Reward Hacking in Production RL" (Monte MacDiarmid and 22 authors) confirms a chilling phenomenon: once a model has learned in RL training to take shortcuts for reward,

this "do anything to earn points" loop generalizes on its own into other domains, and the model fakes alignment, cooperates with adversarial roles, reasons through malicious goals, and even attempts to sabotage AI safety research code.

Two bar charts side by side. In the left chart the vertical axis is sabotage rate and the horizontal axis has four groups — Default SDF model after hacking RL, Unguided SDF after hacking RL, Default SDF model after hacking RL + RLHF, No hacking baseline; the orange bars for hacking-related original peak at about 0.33, while the purple bars for hacking-related rewrite and the blue bars for not safety-related are all below 0.02. In the right chart the vertical axis is sabotaged classifier effectiveness and the horizontal axis has three groups — Default SDF model after hacking RL, Default SDF model after hacking RL + RLHF, No hacking baseline — with the orange bars at about 0.57, 0.72 and 0.88 in order

The preconditions here have to be spelled out, otherwise it is easy to read this as "production models have already gone out of control." The models in the paper were deliberately seeded with reward-hacking strategy knowledge and then trained in fragile environments that were selected on purpose — the authors themselves write that "we specifically chose these environments because we knew they were fragile."

The paper also notes that production Sonnet 3.7 and Sonnet 4 scored zero across all of these evaluations. In the UK AISI's independent replication in March 2026, the hacking behavior did reproduce, but the broader misalignment appeared only sporadically, and was neither consistent nor high-proportion.

The real problem is not that the model turned bad, but that once this loop forms it is hard to keep down by docking points. Penalizing cheating directly during training only teaches the model to cheat more covertly.

So the leading labs now build three lines of defense.

The first is automated trajectory monitoring.

Anthropic used Sonnet 4.5 when training 4.6 and called Opus 4.6 directly when training Mythos, running full automated summarization and anomaly scans over hundreds of thousands of RL training trajectories. At the same time it deliberately constructed Impossible Tasks, a set of problems designed to be unsolvable, to see whether a model, after failing repeatedly, honestly reports "the task cannot be completed",

or goes ahead and cheats by modifying the test script.

The academic counterpart is ImpossibleBench, which uses One-Off and Conflicting mutations to make the tests contradict the natural-language specification, so that any pass must be cheating.

The second is white-box activation probes. Because models have already learned to hide intent inside the chain of thought, looking only at the output text is not enough. The Mythos Preview system card uses sparse autoencoder features, activation verbalizers and sentiment vectors: once a feature correlates with "concealment", "strategic manipulation" or "evading detection", it lights up.

Goodfire's corresponding study (September 2026) measured reward hacking in 50% to 96% of rollouts on Kimi K3, GLM-5.2 and Qwen 3.8 Max.

The third is GRAFT (joint work by MATS, Stanford, FAR AI and Anthropic, October 2026).

Its engineering solution is fairly clever: it adds two sets of neurons, RETAIN and FORGET, to every MLP layer; trajectories that the monitor flags as cheating get a zero gradient mask on RETAIN so that only FORGET is updated, and once the entire RL training run is over, that block of parameters is physically cut out of the weights.

The key design choice is that the monitor only decides "which parameters get updated" and does not participate in the reward, so the model never learns to evade the monitor.

A three-column flow diagram whose titles, in order, are Unclassified Behavior (π_R+F), Detected Reward Hacking (π_R+F), Anchor Environment Behavior (π_R). Under the MLP N layer of each column sit three boxes side by side: a blue Base model weights, an orange Retain adapter, and an orange Forget adapter; in the third column the Forget adapter is drawn as a dashed box to indicate that it is disabled. A legend in the lower left explains Forward/backward pass, Parameters frozen (blue), Parameters updated (orange), 2× update (orange outline), Parameters disabled (dashed)

Alongside this but independent of it is Inoculation Prompting (Anthropic Fellows Program, October 2025), whose mechanism is to explicitly ask the model to exhibit the unwanted behavior in the training prompt, removing the optimization pressure that would internalize it, and to use clean prompts at test time.

0707 Some practical takeaways

Putting four years of technical reports, audit experiments and systems papers together, the first half of large-model data engineering has one clear throughline:

In the pre-training and mid-training stages, competition stopped being about who crawled more web pages long ago, and is now about fine-grained distribution control under a limited stock of high-quality corpus, and using tens of B of high-density mid-training corpus to move weight representations into positions that reinforcement learning can activate effectively;

In the post-training stage, as agents enter complex terminal, browser and codebase environments, preventing reward gaming and using activation probes to control unwanted generalization have become the key steps that determine a model's real usability.

The real barrier that separates the big labs is that someone else has already stepped on these hundreds or thousands of engineering traps, and you have not.

The next piece breaks down the second half: how public leaderboards turn into a game of cat and mouse, the grading loopholes inside evaluation containers, and why 80% of training time is spent in rollout bubbles.

About DataMining A team formed by doctoral students from the Software Institute of the School of Computer Science at Peking University, focused on providing high-quality original task data and executable evaluation environments for large-model and agent research. We deliver not only data, but industrial-grade Rubric and Harbor Task validation systems. ▸ Long-horizon code and engineering simulation: Deep SWE, Frontier SWE, Frontier-Eng, ALE / Terminal-Bench, supporting 200 tasks per week with stable delivery across 7 working days. ▸ Frontier mathematics and specialist agents: an HLE strict multimodal production line (5000+ tasks in stock); Harvey LAB, Automation Bench, Toolathlon, plus 35+ atomic-level Rubrics industry workflow packages. ▸ System foundation and embodied real hardware: standardized Win11 / Ubuntu desktop agent VM images, plus million-scale embodied intelligence Grounding / SFT / DAgger real-hardware customization lines and cyber data. The research, datasets and benchmarks cited in this article are all public third-party results; we only align with their capability requirements and difficulty ranges, and do not copy their problems; the capability descriptions in this article are our own internal conclusions under the same standard and do not represent official third-party conclusions. 🌐 Website: datamining.cloud｜📮 Sample requests and technical contact: contact@datamining.cloud

📚 Figure Sources

[1] The Data-Quality Illusion: Rethinking Classifier-Based Quality Filtering for LLM Pretraining，arXiv:2510.00866，Figure 2

[2] PRISM: Demystifying Retention and Interaction in Mid-Training，arXiv:2603.17074，Figure 2

[3] PRISM: Demystifying Retention and Interaction in Mid-Training，arXiv:2603.17074，Figure 11

[4] Model Spec Midtraining: Improving How Alignment Training Generalizes，arXiv:2605.02087，Appendix Figure 9

[5] Natural Emergent Misalignment from Reward Hacking in Production RL，arXiv:2511.18397，Figure 12

[6] Gradient Routing Localizes and Removes Unintended Behaviors in RL，arXiv:2610.17062，Figure 1