Document order and Lost-in-the-Middle tests

Use a set of order-independent records when the task permits reordering without changing meaning. Moving a paragraph inside a narrative can break references or chronology, changing the question rather than isolating a position effect. Keep the same supporting record, distractors, and answer policy across placements. Record the evidence span’s token offsets and the full prompt length under the actual tokenizer, including instructions and separators. A middle document can still occupy an early token position if other documents have unequal lengths.

Control truncation and output budgets explicitly. If a relevant source falls outside the accepted input after reordering, the result measures missing evidence rather than an internal position bias. Also inspect duplicate answer-bearing records in the distractors and accidental clues in document titles. Fresh random keys help reduce memorized-answer shortcuts, but an exact lookup is still narrower than comparing several documents or integrating a chain of facts. Use synthetic and natural task conditions as complementary evidence rather than treating one as proof of the other.

Report position-specific results and the range between the best and worst placement, not just a pooled mean. A strong average can hide a failure region that matters for the product. Compare model releases on the same base questions and inspect paired successes and failures. Uncertainty estimation should preserve question grouping across placements; counting each moved copy as an independent sample overstates the amount of evidence. Include several prompt templates when template sensitivity is part of the intended claim.

A U-shaped observed profile can motivate hypotheses about primacy, recency, attention, or training, but a performance curve alone does not identify its causal mechanism. Tokenization, query location, document ordering, and training overlap can all affect the comparison. The original study also distinguishes synthetic retrieval and multi-document QA behavior, so a prompt change that fixes one test need not fix another. State exactly which conditions improved before making a claim about general long-context robustness.

Retrieval ranks, chronological sorting, grouping by source, or placing decisive passages near instructions can change final answer quality. Evaluate a proposed ordering policy on held-out questions and inspect whether it creates citation or chronology errors. If a policy uses a learned relevance score, include its own misses; an oracle placement that knows the answer is a diagnostic upper condition, not a deployable router. Preserve source identities when moving text so attribution remains testable and a later error can be traced to its visible evidence.

The function lab supplies a pure quadratic illustration only. Reset, set coefficient to 1 and position to −1, predict y, then Step: one. For a separate run, Reset and set both coefficient to 1 and position to 0 before Step: zero; repeat a fresh Reset with both coefficient 1 and position 1 to obtain one. These are values of y = ax², not measured language-model accuracies or a fitted Lost-in-the-Middle curve. The instrument does not reorder documents or test recall.