Data Mixture: A Zero-Sum Game Under a Fixed Compute Budget (Part 2)

The previous part covered cleaning and generating. This one is about mixing.

The core logic is simple: the total token count is fixed. Give code a little more and every other domain gets a little less. This is not a theoretical concern — it is a reality of every training day.

0101 Where staged release came from

Early on, teams mixed everything once and trained on it end to end. That did not work well. A model's final behavior is strongly shaped by the data it sees at the end of training. While the learning rate is still high, the model learns coarsely; near the end, as the learning rate decays, it learns more finely. That is when the most valuable data has the greatest effect on how the model feels.

SmolLM3 is the typical example:

• Stage 1 (roughly 8T–9T): mostly base web, about 10% code, 3% math, 12% multilingual — cover the knowledge breadth first.
• Stage 2: bring in higher-quality math and code (FineMath4+, MegaMath and others).
• Stage 3 (decay, roughly 1.1T): add high-precision reasoning and instruction data such as OpenMathReasoning and OpenCodeReasoning.

A four-stage stacked area chart; x-axis at 0, 6T, 8T, 10T and 11T training tokens, labeled Stage 1 through Stage 4; legend has four categories — English Web, StarCoderData/Stack-Edu, OWM/InfiMM-WebMath/FineMath, Textbooks; the stages are labeled 90% and 10%, 75% and 20% and 5%, 74% and 16% and 10%, and 58% and 24% and 14% and 4%.

The logic is like studying for an exam. You would not burn all your best past papers on day one and then grind through bad questions for weeks. You cover the fundamentals first and use the last few days on real exam papers to calibrate.

OLMo 2 follows a similar idea: web dominates early on, the late annealing phase switches to the higher-quality Dolmino-Mix, and then model soup is applied. Llama 3 also dynamically upsampled math and topped up newer web data during training.

0202 Annealing ablations: cheap validation for new datasets

Retraining 11T from scratch for every new dataset is far too expensive. So teams now use annealing ablations: take a checkpoint around 7T, run 40% of the original recipe plus 60% of the candidate dataset for 50B tokens, and look at the result. You get an answer in days. MegaMath was validated this way.

This cuts validation cost from "months" to "days", and the signal is strong enough to decide whether a dataset deserves a place in the next stage.

Two line charts labeled GSM8K and MATH AVG; x-axis Training Tokens (B) from 0 to 100, y-axis Accuracy; legend has three curves — OWM, InfiMM-WebMath and Baseline.

0303 Why automated mixture algorithms have not replaced humans

DoReMi and RegMix look elegant on paper: use a small model or a regression fit and let the model learn the weight of each data domain. It sounds far more scientific than trying ratios by hand.

A three-step flow diagram: on the left, seven blue weight bars of different lengths for Wiki, Books, News, Web, Code, Law and Med, labeled Reference domain weights; Step 1 green box reads Train small reference model, Step 2 green box reads Train small proxy model with DRO to get domain weights, Step 3 green box reads Train large language model with reweighted dataset; on the right, optimized bars for the same seven categories and a neural-network sketch.

A four-step diagram: (1) Train small-scale proxy models, a table with percentage combinations across Hacker News, Github and Philpapers (9.5%/35.9%/54.6% and others) and corresponding Target values 5.46, 5.57, 6.07; (2) Fit a regression model using data mixture as source, with a yellow box showing Regression Model, Linear Model and Tree Model; (3) Simulate new data mixtures and predict Target, with a 3D surface plot on the right, x-axis Hacker News (%) and Github (%), color bar Prediction 5.4–6.0; (4) Train a large-scale model on the best mixture, with Predicted Best Target 5.34 at the bottom.

In practice, especially at the trillion-token scale of SmolLM3, the weights these methods learn tend to closely track the natural size distribution of the datasets themselves. Put bluntly, they end up saying "use more of whatever you have" — and they still lose to carefully designed human ablations.

The reasons are practical: proxy and target models differ in scale, training length and evaluation metrics; cross-domain interactions and repetition effects are hard for a simple regression to capture.

Newer work (for example RegMix-D) uses the entire loss trajectory for dynamic mixing, and some teams use model merging to lower search cost. But at real scale, the core recipe still comes from "run a set of experiments → check the key scores → adjust the ratios → run another set".

0404 Hidden engineering traps

Even with the mixture settled, data-loading details can wreck training.

The classic case is shuffling. Documents are shuffled, but sequences inside a batch are still read sequentially from the same document. Short documents are harmless; code files are often very long. If one long file is low quality, it can fill an entire batch. The model then eats a batch of homogeneous junk and the loss suddenly spikes.

Two stacked loss charts: the top y-axis is loss (2.0–3.0), the bottom y-axis is L2 norm of the gradient (0.0–3.0), x-axis training steps to 600000; legend shows blue for OLMo 0424 7B and orange for OLMo 2 1124 7B; the blue curves show many vertical spikes in both charts while the orange curves are smooth.

The fix is plain: shuffle tokenized sequences offline in advance, change the seed every epoch, and drop documents with unusually many repeated internal n-grams. OLMo 2 confirmed that similar treatment clearly reduces spike frequency.

Two gradient-norm curves side by side; y-axis L2 norm of gradient on a log scale from 10⁻¹ to 10¹, x-axis step from 0 to 17500; left chart legend without repetition filter, right chart legend with repetition filter; the left chart has visibly more vertical spikes in the second half.

Other couplings matter too: deduplication and contamination checks must finish before mixing; quality filtering and the mixture influence each other; and the data upsampling used for long-context extension needs its own validation.

0505 A practical note to close

The ceiling on data mixing has never been "what the target wants", but "how much unique high-quality data you actually hold". Blindly raising the share of scarce data only makes the model memorize.

The most reliable path today is still: clear inventory constraints + a staged curriculum + fine-grained data in the final phase + many small-scale ablation checks. Automated algorithms can help, but they do not replace experienced judgment yet.

Cleaning subtracts, generating adds, and mixing decides what the model studies each day under a fixed compute budget. Cut corners on any of the three and the final result suffers. Public reports usually give only the final recipe; what is genuinely valuable is reconstructing the failed experiments and the engineering constraints behind it.

If you are doing similar work, design your mixture experiments as a reusable small-scale ablation framework with explicit inventory, repetition thresholds and key evaluation metrics. That is much closer to what actually works today than chasing a one-click optimal mixture algorithm.

About DataMining Founded by a team of PhD students from the Software Institute, School of Computer Science, Peking University, we build original high-quality task data and executable evaluation environments for large models and agents. We deliver not only data, but industrial-grade Rubric and Harbor Task verification systems. ▸ Long-horizon code and engineering simulation: Deep SWE, Frontier SWE, Frontier-Eng, ALE / Terminal-Bench and more, supporting 200 tasks per week with stable delivery in 7 working days. ▸ Extreme math & science and professional Agents: a strict multimodal HLE production line (5,000+ tasks in stock); Harvey LAB, Automation Bench, Toolathlon, plus 35+ atomic-level Rubrics industry workflow packs. ▸ System infrastructure and embodied real machines: standardized Win11 / Ubuntu desktop Agent VM images, plus million-scale embodied-intelligence Grounding / SFT / DAgger real-machine custom production lines and cyber data. All research, datasets and benchmarks cited here are public third-party work; we only align with their capability requirements and difficulty ranges and do not copy their tasks. Capability descriptions in this article are our internal same-criteria conclusions and do not represent official third-party conclusions. 🌐 Website: datamining.cloud | 📮 Sample requests and technical contact: contact@datamining.cloud

📚 Figure Sources

[1] SmolLM2: When Smol Goes Big — Data-Centric Training of a Small Language Model, arXiv:2502.02737, Figure 2

[2] SmolLM2, arXiv:2502.02737, Figure 5

[3] DoReMi, arXiv:2305.10429, Figure 1

[4] RegMix, arXiv:2407.01492, Figure 3

[5] 2 OLMo 2 Furious, arXiv:2501.00656, Figure 2

[6] 2 OLMo 2 Furious, arXiv:2501.00656, Figure 3