Example sampling is not token weighting

Task probabilities can govern which example is sampled, which conversation is sampled, or which token contributes to a loss. Those are different objectives. Equal-example means weight a short answer and a long answer equally after their internal normalization;global token means give more mass to examples with more selected targets. Padding and masked prompt tokens do not belong in a target-token count. Track effective counts after formatting and truncation, not only the original dataset's row table.

A large dataset can dominate without explicit resampling, while oversampling a small dataset repeats its few examples more often. That can help coverage or amplify artifacts and memorization. Task labels may also hide source clusters, language differences, and response-format styles. Keep provenance and group-aware splits alongside the mixture. A mixture change is a new training intervention that needs controlled comparison, rather than a harmless input shuffle assumed to improve every task.

One can sample tasks at chosen rates and also apply explicit per-task loss weights. Document both, and verify that trainer reductions do not accidentally undo the intended scheme. Averaging local batch means with unequal target counts can change weights again. If curriculum or temperature-based sampling varies over training, the final model has seen a time-dependent mixture;report exposures and steps by stage rather than one nominal starting ratio.

The best mixture depends on held-out use cases and capability-retention requirements. A domain-specific instruction corpus may improve its target format while harming general tasks or changing refusal and verbosity behavior. Evaluate subgroups and failure severity, not merely a grand average that lets one easy task mask another's regression. Keep validation for mixture selection and avoid repeatedly using the final test to pick the weights. No algebraic balancing rule guarantees downstream quality.

Record sampled examples, unique source groups, selected target tokens, and optimizer steps for every task. Compare those realized values with the planned mixture and inspect records that were truncated or rejected. A long-answer task can lose target tokens disproportionately at a context limit, changing its effective weight. Repeated examples do not create independent evidence of generalization. Use multiple seeds or justified uncertainty estimates when claiming a small benefit from a mixture change.

The probability lab models task sampling only. Reset and set both p=0.5, n=20, predict ten selections of a named task on average, then inspect. Reset and re-set p=0.8, n=20 to predict sixteen. Its independent Bernoulli draws contain no response lengths, loss weights, or training dynamics. Play highlights possible counts rather than sampling an actual dataset. Explain why equal task-selection counts in this diagram cannot establish equal supervised-token contribution or equal learning progress.