Track cumulative exposure when the domain mixture changes

A probability over documents is not generally the same as a probability over valid tokens. Domains with longer documents can contribute more targets per sampled item, and packing or truncation can change that contribution. Define whether weights control documents, shards, sequences, or token mass, then measure realized valid-token shares. If the sampler draws documents with replacement, record the replacement policy and deduplication unit. An announced weight of twenty percent should not be described as twenty percent of training tokens without that accounting.

Also define when the weights change: successful optimizer update, consumed token count, accepted trajectory tokens, or another clock. Store sampler state and current phase with checkpoints so recovery does not accidentally restart the early curriculum. Adaptive changes based on development metrics should record the observed evidence and decision rule. That adaptation is part of the experiment, and repeated selection on development results does not leave those results equivalent to an untouched final test.

Suppose the early phase contains thirty million targets and the later phase seventy million. A domain's average share is weighted by those counts, not the arithmetic mean of its two configured percentages. Compare the resulting exposures with the available unique data. Repeated exposure can be intentional, but four nominal passes over a small domain do not mean every example is seen exactly four times under replacement sampling. The actual occurrence distribution depends on the sampler and document lengths.

Hoffmann and colleagues' compute-optimal study connects model size and training-token budget through empirical experiments and fitted assumptions. DoReMi studies data-mixture optimization in a particular proxy-model method. Neither establishes one universal token-per-parameter ratio or guarantees that any easy-to-hard curriculum improves every model. Use controlled ablations at matched declared budgets, record total compute and realized tokens, and examine domain regressions. A favorable aggregate can conceal loss of coverage when the mixture repeatedly favors already strong domains.

Compare the planned mixture schedule with a fixed-mixture baseline under the same architecture, tokenizer, objective, optimizer policy, and declared resource budget. Log realized source counts, rejected examples, sampler exhaustion, and replacements. Changes in sequence length affect both packing and context, so they can change more than domain balance. Preserve the exact policy rather than claiming that the curriculum alone caused an improvement when several dimensions changed together.

The sampling lab quantifies realized-count variation for a deliberately simpler independent assignment process. Reset, set both p=0.2 and n=100, predict a mean proportion 0.2 and variance 0.0016, then Play or Step. Reset and set both p=0.6,n=100 for mean 0.6 and variance 0.0024. Its exact binomial bars are not sampled documents and do not implement a changing curriculum, replacement over a finite corpus, or token-length effects. They show why an intended probability and an observed finite count should be logged separately.