Positional extrapolation and scaling tests

RoPE rotates paired query and key coordinates by position-dependent angles. Within one pair and one frequency, the query-key dot product depends on the difference between their positions. That identity explains a relative-position mechanism; it does not show that learned content interactions remain useful outside the training distribution. Individual coordinate-pair contributions oscillate and need not decrease monotonically with distance. Aggregate claims about decay or generalization require their stated assumptions rather than being inferred from one cosine component.

Identify the checkpoint’s original position convention, tokenizer, training lengths, and implementation before altering it. Query and key rotation, cached-key representation, and serving configuration must agree. A scale change applied inconsistently during cached decoding can introduce an implementation bug distinct from a model’s extrapolation weakness. Reproduce a small uncached versus cached comparison where possible, then test beyond the original range. Equal tensor shapes and a successful forward call are compatibility checks, not evidence that distant facts were actually used.

Uniform Position Interpolation maps a longer index range into the original range. It compresses relative distances in the positional coordinates rather than deleting tokens. The model still receives the longer sequence, with its memory and attention costs, and may need adaptation to distinguish newly compressed positions. YaRN modifies aspects of frequency-dependent scaling and attention scaling rather than being equivalent to every uniform index division. Pin the exact method and configuration when attributing a result to a position-extension recipe.

Compare a scaling-only run, an adapted model, and the original checkpoint where the inputs are legal. Keep training token budget, data mixture, and evaluation protocol visible; a stronger result after more training is not isolated evidence for one positional formula. Also test the original short lengths because scaling can alter behavior there. Report task quality by evidence position and separation as well as sequence length. A successful single-needle retrieval run does not establish multi-hop reasoning, summarization fidelity, or robust performance on every language.

Construct fresh examples with controlled evidence location, distractors, and dependency distance, using the actual tokenizer to verify placement. Include tasks requiring several separated facts and checks for unsupported inference, not only copying an exact key. Keep held-out source documents and generated templates separate from adaptation data. Repeatedly tuning against one visible synthetic needle can overfit a narrow format. A credible extension claim names the tested length range, workloads, prompt policy, and uncertainty instead of treating the configured maximum as the measured capability.

The vector lab illustrates one rotary phase only. Reset, set angle to 120 degrees and magnitude to 1, predict the horizontal projection, then Step: negative one half. Compare a fresh Reset with both angle at 30 degrees and magnitude at 1 before Step: approximately 0.8660. Dividing this phase by four resembles one uniform scaling operation, but the instrument has no multiple RoPE frequencies, learned attention, cached keys, training, or long-context task evaluation.