Compare models on matched cases and preserve the pairing

For each eligible case, obtain scores from model A and model B under the declared resource, retrieval, and scoring contracts. Define a positive difference as improvement in the direction relevant to the metric. The mean of paired differences estimates the chosen population contrast when the collection and measurement assumptions support it. Its sampling variance involves the covariance between model scores, which an analysis treating the two score lists as unrelated ignores. Pairing cannot fix a change in prompts, tool permissions, or annotation rules applied only to one model.

Use a shared case identifier to join outputs and record missing or failed evaluations before computing the contrast. Restricting to cases where both systems succeeded can create a different conditional estimand, so choose and report the missing-outcome policy. If several outputs come from the same user or source, retain that cluster in the paired bootstrap or another appropriate procedure. The same selected cases and clusters must appear for both systems within each resample. For corpus-level metrics, recompute the metric rather than assume it equals the mean of per-example scores.

For binary success scores, cases where both models pass or both fail have zero difference. Let W count B-only successes and L count A-only successes. Under an independent-pair null with equal probabilities of these two discordant outcomes, conditioning on W+L gives a binomial count of W with success probability one half. An exact fixed-sample tail calculation can test that null. It is a test of a specified contrast under assumptions, not a posterior probability that B is better and not a guarantee that the improvement exceeds a practical threshold.

State the alternative and p-value convention before inspecting the outcome. A one-sided improvement test and a two-sided difference test answer different questions. Ties, abstentions, and parsing failures need a declared policy; silently dropping them after seeing which model benefits can bias the analysis. For continuous or ordinal outcomes, choose a method suited to the score and design instead of forcing every comparison into a binary win count. Koehn's paired-bootstrap discussion illustrates the importance of resampling matched outputs when comparing evaluation systems on the same test cases.

If a team tries many prompts and reports the model pair with the smallest test p-value, the single fixed-comparison calculation no longer describes the full selection procedure. Preserve a final confirmation role or use an inference design that accounts for the planned comparisons and adaptations. Repeatedly recomputing an ordinary fixed-sample test and stopping at significance also changes its error behavior. Time-uniform methods have their own assumptions and boundaries; they are not obtained by relabeling a fixed-sample interval as safe for continuous monitoring.

The probability lab represents only the exact conditional null in a small example. Reset, set both p=0.5 and n=10, predict the upper tail from eight through ten wins, then inspect the bars and add them. The tail is 56/1024=0.0546875. A symmetric doubled-tail two-sided value for this observation is 0.109375. The instrument has no pairing, model outputs, clustering, or automatic p-value computation. It displays the binomial masses to be summed under the stipulated null, while the actual evaluation must justify why that null applies.