Mixture weights allocate training exposure rather than create new information

A domain can mean source family, genre, language, modality, or another documented grouping. The categories should have stable membership or an explicit overlap rule. If one document belongs to both technical and English collections, simply adding their counts can double-count available text and obscure effective exposure. Pin extraction, tokenizer, deduplication, and domain assignments before measuring sizes. Uniform document sampling favors long documents in token volume unless length is accounted for; uniform sequence sampling has its own packing and truncation effects. Compare intended weights with observed emitted tokens after loader filtering.

A temperature-style size rule can interpolate between uniform domain weights and natural token proportions, but it is a policy family rather than an optimization proof. For positive domain sizes, exponent zero gives equal weights and exponent one gives size-proportional weights. Changing the exponent changes exposure to both rare content and repeated examples. If some domains are excluded for source or sensitive-data reasons, a sampling rule cannot legitimately restore them merely to satisfy a target proportion. Apply mixture optimization only within the documented admitted candidate corpus.

For a fixed emitted-token budget, multiply the budget by each domain probability to obtain its expected allocation. Divide by that domain's available tokens to express an expected number of passes when repetition or replacement is allowed. This average does not guarantee that every document is sampled equally, particularly with variable lengths or clustered duplicates. A domain that is upweighted fivefold may offer valuable rare content, but the sampler has not manufactured five independent copies of its information. Audit repetition and held-out loss rather than equating exposure with diversity.

DoReMi learns domain weights using a small proxy model and then trains a larger model on the resulting mixture; its reported experiments show why weight selection can be more principled than preserving raw source proportions. The result is evidence for that method in its studied settings, not a guarantee that transferred weights are optimal for every architecture, tokenizer, budget, or target application. Compare candidate mixtures under controlled training conditions and protected evaluations. Report the definition of the domains and any proxy-to-target mismatch, since different partitions can create very different optimization problems.

If importance weighting is used to estimate a target distribution from another sampling distribution, every target-supported region needs positive sampling support and well-defined ratios. Very large weights can increase estimator variance and concentrate learning on few examples. Resampling and weighting loss terms can share an expected objective under suitable conditions while producing different minibatch variance and optimization behavior. Distinguish those implementations in the manifest. Keep domain validation sets separate from corpus duplicates and benchmark tuning, and track per-domain outcomes so an aggregate gain does not conceal a serious coverage loss.

The probability lab models only independent domain choices. Reset, explicitly set BOTH B-domain probability p=0.1 and n=20, predict mean two B draws and variance 1.8, then Play or Step. Reset and re-set BOTH p=0.5 and n=20; predict mean ten and variance five. Edits retain t, and playback highlights the exact distribution rather than sampling training batches. The toy has no documents, variable token lengths, repetition, optimization, or DoReMi proxy model. Its p corresponds to a stipulated draw-level mixture only after that unit is defined.