Pack documents without changing who can attend to whom

Assign each real input position a document identifier and an ordinal within its packed storage row. For independent documents, a query may read a key only when that key is in the same document and is not later than the query. Self-attention to the current input position is allowed because that input predicts the next token. This is different from allowing the target token to enter its own context. The shifted-label fixture must establish which positions carry inputs and which carry targets.

Write the allowed relation as an abstract predicate before translating it into a framework API. In PyTorch 2.9 scaled_dot_product_attention, a Boolean True means participation, while other APIs can use a different Boolean convention. A float bias can instead add zero to allowed scores and negative infinity to forbidden scores under supported semantics. Check the actual selected implementation's broadcast shapes and supported combinations rather than assuming an is_causal flag also implements document boundaries.

A chosen positional encoding may reset indices at each document start or use another documented convention. Resetting indices does not remove an attention edge. Conversely, blocking cross-document attention does not by itself establish the desired position numbers. Preserve both policies when comparing packed with separate execution, especially for learned absolute positions or relative-position mechanisms whose behavior depends on the indices. A packing implementation should expose the generated document IDs, position IDs, and target eligibility in a small inspectable fixture.

For [BOS,a,b,EOS] followed by [BOS,c,EOS], seven real input tokens occupy the row. Under the declared independent policy, only five next-token targets are scored: three from the first document and two from the second. Padding the row to eight positions creates no additional language-model target. Avoid evaluating undefined padding-query attention rows, or use a framework-supported safe treatment and exclude their outputs as required. A finite final loss can conceal incorrect attention if those intermediate rows are never checked.

Turn off dropout and other stochastic effects for a numerical equivalence fixture, then compare each document's logits at corresponding scored positions using the same model and position policy. Also alter one packed document while keeping its neighbor fixed; independent masks should prevent the neighbor's predictions from changing beyond the declared numerical tolerance. Check a deliberately broken cross-document mask as a positive control so the fixture demonstrates that it can detect the forbidden information path.

The modular lab illustrates only a local-position counter for fixed-length documents. Reset, set both modulus m=4 and additive step a=1, predict positions zero, one, two, three, zero, then take four Steps. Reset and set both m=3,a=1 for a different fixed length. Each half-second Step advances one integer addition. The lab neither builds attention matrices nor supports mixed document lengths, padding queries, token labels, or a Transformer; its wraparound shows why a position reset is one small component of packing correctness.