Data Mixture Is a Zero-Sum Game Under a Fixed Compute Budget
54B maths tokens, 628B multilingual against 5.1T English, and why 25% code hurts. Inventory limits, staged curricula, anneal ablations and the limits of automatic mixing.
Before pretraining, teams already clean their data — deduplication, contamination removal, low-quality web filtering — and synthesise data with automatic verifiers. Those steps answer "what usable data do we have?". What determines the final capability profile is a different question: with a fixed total token budget, what share does each domain get?
That is data mixture, and it is a zero-sum allocation problem. One extra point for maths is one point taken from somewhere else. The ceiling is set not by a wishlist of capabilities but by three realities: how much non-duplicate, high-quality stock actually exists; whether knowledge transfers across domains or repeated exposure saturates; and how much influence late-stage data retains once the learning rate has decayed.
01Why the ceiling is low: quality stock is scarce
Maths. FineMath3+ and InfiWebMath3+ together contain only about 54B tokens. In an 8–9T-token early run, a maths share above 3% means the same problems are seen more than five times. The model then memorises question surfaces and answer distributions — "C is usually right" — rather than learning to reason.
Multilingual. High-quality corpora for French, Spanish, German, Italian and Portuguese total roughly 628B tokens against 5.1T for English, nearly an order of magnitude apart. Ablations put the balance around 12% multilingual: enough coverage without eroding English capability.
Code. Claims that a high code share lifts reasoning and world knowledge do not hold universally. In 3B-scale ablations, 25% code visibly depresses HellaSwag, ARC and MMLU; 10–12% is the stable range.
These are not priorities. They are inventory constraints plus the cost of repetition.
Two public recipes are useful reference points. Llama 3 settled near 50% general knowledge, 25% maths and reasoning, 17% code, 8% multilingual, with a knowledge classifier downsampling over-represented categories and scaling-law experiments to tune the rest. OLMo 2 runs an explicit two-stage curriculum: broad web data first (OLMo-Mix-1124), then a higher-quality anneal mix (Dolmino-Mix: quality web, academic text, QA, maths, synthetic data), followed by model soup. SmolLM3 points the same direction, with ratios shifting by source quality and target.
Synthetic data is not a loophole either: it still consumes budget and still saturates.
02Stage the curriculum: foundations first, fine detail last
High-quality, carefully curated data has more leverage late in training, during the anneal phase when the learning rate is low and the model is settling. A cheap way to test a candidate mix is anneal ablation: run a short late-stage anneal on a small compute budget with different mixtures, then compare downstream scores. It is far cheaper than full pretraining and is how several public recipes were actually derived.
03Automatic mixture optimisation is not a replacement yet
DoReMi-style reweighting and other automatic methods are genuinely useful as reference signals. At industrial scale, however, they do not replace systematic ablations and cross-domain judgement: the objective is multi-dimensional, the evaluation suite is expensive, and the optimum moves with the model size and the token budget. The competitive advantage remains the experiment framework, not the one-shot optimiser.
04A correct recipe can still break in the loader
Even with the right ratios, engineering details can wreck a run: shuffling granularity, domain packing, repeated epochs on small high-quality sets, sequence length policy, and the exact learning-rate schedule at the anneal boundary. A mixture is not a CSV of percentages; it is a schedule executed by a data loader.
05What teams should take away
Treat cleaning, synthesis and mixing as three separate steps with separate failure modes. Build a reusable small-scale ablation harness before committing the main run. Record inventory, repetition thresholds and evaluation metrics per stage. Public technical reports usually publish only the final recipe; the valuable playbook is the set of failed paths and constraints that produced it.
About DataMining Founded by PhD researchers from the Institute of Software, School of Computer Science, Peking University, DataMining builds original task data and executable evaluation environments for frontier models and agents, with reproducible rubric specifications and bootable Harbor Task verifiers. Benchmarks named here are public third-party benchmarks; we align capability requirements without copying tasks. Visit datamining.cloud or write to contact@datamining.cloud.