Control presentation and length without erasing real task differences

Randomize which candidate appears in each display slot and store the mapping back to the candidate's stable identity. Otherwise a label meaning left wins can accidentally become a label favoring whichever generator was always placed left. For a controlled order check, evaluate the same prompt and answer pair in both orientations under a recorded judge protocol, resetting conversation context and accounting for stochastic repetition. Preserve the paired records: two orientations of one prompt are related observations, not two independent new prompts. Conceal irrelevant model labels when the experiment is intended to assess response content.

Measure both candidate win rates by position and within-pair consistency. A judge can switch content preference after swapping slots, or a consistent content preference can coexist with other population-level biases. Report ties, invalid outputs, and parsing failures rather than silently dropping inconvenient decisions. MT-Bench and later position-bias research document empirical presentation effects in particular judge/task settings. They motivate testing your own protocol, not a universal correction factor that applies to every human annotator, automated judge, or response quality gap.

A longer response may contain more reasoning, necessary qualifications, or irrelevant repetition. Raw correlation between reward and length cannot distinguish those causes. Analyze matched tasks and quality evidence, inspect controlled rewrites where content is checked, and stratify by response-length differences. Shortening text can remove essential information, so a length-matched rewrite is not automatically a pure intervention on length alone. For training, ensure score reduction is identified: sequence log-probability sums and token means have different length behavior, and swapping one for the other changes the stated objective rather than merely removing a harmless display artifact.

Length-controlled AlpacaEval proposes a statistical adjustment for a particular evaluator setting. Such an adjusted score estimates a chosen comparison under modeling assumptions, not an observed universal utility ranking. Unmeasured content changes and model misspecification can remain. Report the raw and adjusted estimands, data coverage, and relevant held-out checks instead of selecting whichever favors a release. An evaluation whose task explicitly requires detailed explanation may value some extra length; a terse-answer task may not. The rubric should resolve that distinction before looking at candidate names or final scores.

Choose the order, repetition, and length-control analyses before selecting the winning checkpoint. Use prompt-aware paired or grouped uncertainty where appropriate, and keep a held-out confirmation set if the protocol is tuned after seeing results. A single swapped judgment can be noisy, so distinguish consistent effects from random variation without discarding difficult examples. Record judge weights or version, prompt template, decoding settings, rubric, and parser. A changed template can alter the judgment process even if the named judge stays the same.

The probability lab models independent randomized slot assignment only. Reset, set both p=0.5 and n=20, predict the number assigned to the first slot, then inspect. The exact count distribution has mean ten and variance five, so random assignment need not yield exactly ten first-slot presentations in a small group. Playback highlights mathematical outcomes rather than running judges. This instrument has no response length, task rubric, candidate quality, or position preference. It helps separate assignment variability from the actual paired judgments needed to measure a position effect.