Long-context training mixture design

A training mixture can be specified by documents, sequences, input tokens, supervised target tokens, or optimizer updates. These are different units. A long input with a short supervised answer may consume substantial compute while contributing relatively few unmasked loss terms. State whether batches sample examples uniformly or target token proportions, and how losses are normalized. Otherwise an apparent benefit from a long-data mixture can come from changing the effective objective or training exposure rather than improving long-distance information use.

Character and word lengths are not interchangeable with model-token lengths, especially across languages. Measure with the deployed tokenizer and record truncation. A long document cut into short pieces no longer supplies the same long-range dependencies, while concatenating unrelated short pieces produces a long tensor without necessarily creating useful coherent context. Include tasks that demand distant relationships when that is the intended capability. Mere exposure to high position indices does not establish that the model learned to integrate several separated facts.

Packing can reduce padding waste by placing several examples in one tensor, but attention masks and position policies must preserve the intended example boundaries. If unrelated examples are meant to be independent, accidental cross-example attention can expose targets or create spurious dependencies. Loss masking must distinguish instructions, context, and supervised answers according to the objective. LongAlign discusses packing with attention isolation and loss weighting; those details are part of the training method rather than an incidental throughput optimization.

For full dense attention, pairwise interaction work grows roughly quadratically with sequence length before accounting for projections, feed-forward layers, and implementation effects. Thus long examples can dominate the attention component even when their token share is modest. Flash-style kernels can avoid materializing the full score matrix and change IO behavior, but do not make the mathematical full-attention pair count linear. Measure actual training time and memory under the chosen architecture instead of turning a pair-count proxy into a hardware benchmark.

Split source documents, near duplicates, generated templates, and answer-bearing records before forming long sequences. A document split after overlapping windows can place almost identical evidence into both training and evaluation. Synthetic needles also need fresh values and held-out generation patterns; repeatedly testing the same template can reward memorized format behavior. Record corpus versions, deduplication rules, mixture weights, lengths, and adaptation tokens so a later result can distinguish data effects from a positional or architecture change.

The probability lab illustrates only a declared sequence-sampling assumption. Reset, set long-sequence probability to 0.2 and trial count to 10, predict the expected long count, then Step: two. Compare a fresh Reset with both probability at 0.2 and trials at 30 before Step: six. It displays exact binomial masses under independent equal-probability draws, not actual batches, token exposure, quadratic attention work, or data leakage. Use the separate accounting equations to translate a specified mixture into those other units.