Pack tokens without changing which tokens can interact

For a padded batch, useful-token utilization is the number of real tokens divided by all allocated positions, using a consistent treatment of required separators and special tokens. State whether loss-bearing tokens or all input tokens are being counted; prompts may be real inputs while their labels are intentionally masked. Padding waste is one minus the chosen utilization. Bucketing similar lengths can reduce padding, while packing can place several shorter examples in one container. Neither approach guarantees a particular speedup because attention kernels, masks, memory layout, and the model's token-level work still determine cost.

Packing is a constrained placement problem, not merely sorting every batch by length. Choose the container capacity, rules for splitting long examples, separators, and whether order affects sampling. Retain mappings from packed positions back to source identity and original token position. Dropping an inconvenient remainder or repeatedly preferring short examples can change the experiment's visitation distribution. Measure both utilization and tokens contributing to the objective. An increase in allocated tokens processed per second can hide a decrease in useful training tokens if much of the new work remains padding or carries ignored labels.

If the intended task treats examples independently, attention must prevent tokens from reading another example's content. A block-diagonal causal rule permits only earlier or current positions in the same example. Boundary-aware target construction must also prevent an unintended prediction of the next example's first token from the previous example's last token. Position identifiers and any segment-dependent operations must follow the model's intended semantics. The packing paper by Krell and colleagues studies avoiding cross-contamination; its equivalence argument is tied to its model and masking construction, not every architecture or arbitrary concatenation.

Some training objectives intentionally treat concatenated documents as a continuous stream; those are different declared semantics, not automatically incorrect. Decide which experiment is intended before choosing masks. Recheck loss normalization as well: averaging each sequence equally differs from averaging all loss-bearing tokens equally when lengths vary. Packing should not silently change either weighting. Positional encoding, sequence-level heads, dropout behavior, and batch-dependent operations can introduce additional differences even when attention boundaries look right. Use small reference examples to compare outputs and losses under the supported implementation and identify the precise equivalence claim you can justify.

A dense attention implementation may still perform work over a whole packed container even when the mask forbids cross-example attention. A compatible variable-length or block-aware kernel can exploit structure, but its performance and correctness need separate checks. Packing also consumes CPU time, metadata, and potentially a staging buffer while it finds compatible lengths. Include those costs and rank balance in the end-to-end comparison. Report useful tokens per second, step latency, and the objective's actual normalization rather than claim that doubling utilization necessarily halves training time.

The function lab illustrates one limited cost scaling: for each separate view, Reset, set both a=1 and x to 1 or 2, predict y=ax², then inspect. Doubling this abstract length ratio changes the quadratic value from one to four. The instrument is not an attention implementation or a bin-packing solver, and real model cost includes nonquadratic operations and kernel effects. Use it to see why a more populated longer dense container can have a different compute bill from several independent shorter containers, then measure the actual supported masked kernel.