Positional representations and rotary embeddings

For a sequence of length L and model width d, additive position embeddings provide an L by d table or function that is combined with token embeddings. Relative biases instead modify attention scores according to displacement or another positional relation. Rotary embeddings transform query and key features before their dot product. These choices are not interchangeable: they inject position at different points and create different extrapolation and parameter-sharing behavior.

A causal mask already distinguishes allowed past positions from future positions, so a masked attention architecture is not fully permutation equivariant in the same way as unrestricted attention. Nevertheless, a mask is not a general substitute for encoding distances and order. A position convention must also remain consistent between training and cached generation. Reusing cached keys with incorrect position indices can produce valid tensor shapes while computing the wrong attention scores.

A two-dimensional rotation preserves vector length. Rotating a query at position m and a key at position n yields a dot product containing the product of the transpose of one rotation with the other. That product is a rotation through their angle difference. Consequently the positional part depends on n minus m, while the score still depends on both content vectors. Different coordinate pairs provide different angular scales, allowing several distance-sensitive patterns to coexist.

Apply the same pairing layout and frequency convention to queries and keys. Some implementations pair neighboring coordinates and others pair corresponding coordinates from two halves; the weight layout must match the convention. Scaling by the square root of head dimension remains a separate part of scaled dot-product attention. Rotary embeddings do not change the L by L score matrix's basic dense attention complexity, and they do not directly rotate values in the usual formulation.

A formula can be evaluated at positions larger than those seen in training, but that does not prove useful behavior there. Large displacements create unfamiliar phase combinations, and finite-dimensional periodic features can make different distances difficult to distinguish. Frequency rescaling or interpolation changes those phases and therefore changes the model's behavior. Evaluate such extensions against a held-out long-context task, not simply whether inference avoids an indexing exception.

Measure retrieval accuracy as a function of both context length and target position. Include multiple distractors and tests whose answers cannot be inferred from a short prefix. Keep context extension methods, fine-tuning data, attention implementation, and memory budget explicit. A longer context also increases key-value cache storage and prefill compute even if positional features themselves are cheap. Numerical precision in forming angles and trigonometric values can matter at large indices, so verify implementations against a higher-precision reference.