Relevance judgments and pooling bias

A document can be judged relevant, judged nonrelevant, or unjudged. Traditional evaluation implementations often treat unjudged documents as nonrelevant for a metric calculation, but that convention does not turn missing evidence into a human verdict. A newer retriever can return useful material absent from an older pool, making the measured result sensitive to coverage. Inspect the fraction of judged documents at the actual cutoff and the distribution of new candidates, rather than assuming a metric value alone establishes the quality of every novel result.

Define relevance before collecting more judgments: the required answer, supporting evidence, acceptable date, granularity, and partial-relevance levels should be clear. Different assessors can disagree, particularly for ambiguous or multi-intent queries. Record disagreement and adjudication rather than silently collapsing every uncertainty into zero. A positive passage judgment does not necessarily mean the entire source document supports every claim, and an exact answer string does not prove contextual support. The evaluation unit must match what the deployed retriever returns.

If all judged binary labels are accepted as correct, unknown labels create simple bounds on precision at a fixed cutoff. Those bounds are a sensitivity interval, not a statistical confidence interval: they allow every possible assignment of the unjudged items. Recall is more difficult because the number of relevant documents outside the returned list can also be unknown. Calling the fraction of known relevant items retrieved true collection recall hides that missing denominator. State whether a reported recall is conditional on the available judgments.

Pooling is not automatically invalid. Broad, diverse pools have supported stable comparisons in many studied collections, and the magnitude of bias depends on the task, depth, and new system’s behavior. The concern is empirical rather than a reason to dismiss every pooled benchmark. Compare the original evaluation with supplementary blinded judgments for inspected differences, include outputs from competing methods, and sample beyond one model’s favorite results. A selective labeling exercise designed only to vindicate one system can create another bias instead of correcting the first.

Log corpus and query versions, pool contributors and cutoffs, assessor instructions, blinding, and adjudication policy. Keep training labels and evaluation judgments separate, and avoid tuning repeatedly against a small newly labeled disagreement set without a fresh holdout. Report both the ranking metric and coverage evidence, especially for underrepresented query classes. When a gain disappears after broader judging, inspect whether the earlier result came from missing labels or whether the new protocol itself changed the relevance definition.

The probability lab is a sensitivity assumption, not a labeling machine. Reset, set probability to 0.5 and trials to 2, predict the expected number of relevant items among two hypothetical unknowns, then Step: one. Compare a fresh Reset with both probability at 0.8 and trials at 2 before Step: 1.6. This assumes independent common-probability labels that are not known in reality. It does not judge documents, estimate assessor reliability, or narrow the deterministic precision bounds without additional evidence.