Report agreement with its eligible population and uncertainty

For two binary judgments on the same pair, raw agreement can mean the fraction of eligible doubly annotated pairs with matching preferred-candidate identities after accounting for display order. If ties and abstentions exist, decide whether to model them as outcomes, exclude them under an explicit rule, or report several statistics. Preserve their counts and reasons. Excluding pairs that either annotator found ambiguous changes the population being summarized; it can raise conditional agreement while hiding the difficult portion. Missing a second annotation is not automatically disagreement and must not silently enter the denominator.

Aggregation across annotators needs the unit stated as well. Pairwise agreement among several annotators counts different opportunities from agreement of each vote with a majority that includes that same vote. The latter can have a built-in dependence. Report per-task and relevant population slices, since a pooled percentage may conceal systematic disagreements about desired behavior. A chance-corrected coefficient, if used, introduces its own marginal-label model and does not turn agreement into truth. Keep the raw count table available so readers can reconstruct the statistic rather than infer the denominator from a percentage label.

Two annotators can agree because both understand the task, because both follow the same misleading style cue, or because the rubric leaves no room for a relevant alternative. A simple latent-error model illustrates the limitation: if both independently flip the same binary truth with error probability ε, agreement is (1−ε)²+ε². That formula is conditional on a shared truth, identical rates, and independent errors. Real preferences may be heterogeneous or context dependent, and common training or source cues can correlate errors. Do not invert observed agreement into an accuracy claim without defending those extra assumptions.

Repeat annotations and expert adjudication can help diagnose causes, but adjudication is another identified process rather than universal ground truth. Separate stable differences in values or task interpretation from accidental misclicks and missing expertise. Version rubric clarifications and retain previous labels so a change in instructions is distinguishable from model improvement. Original human-preference research includes explicit handling of ties, inability to compare, and noisy labels. Those mechanisms motivate careful collection design; their particular noise assumptions do not certify the error process of a new annotator population.

Under independent identically distributed eligible-pair agreement indicators, a plug-in standard error is sqrt(a_hat(1−a_hat)/N). Proportion intervals need a method appropriate to sample size and boundaries; a naive symmetric interval is only a large-sample approximation and can behave poorly. Shared prompts, annotators, or sessions can invalidate the independent-pair calculation. Use a design-aware analysis, such as a justified grouped resampling scheme, and state which population the uncertainty covers. More votes from the same narrow task do not automatically provide the information of equally many independent new prompt contexts.

The sampling lab isolates the independent model. For each separate view, Reset, set both p=0.8 and sample size n to 25 or 100, predict the sample-mean variance, then inspect. The exact binomial sample-mean variances are 0.0064 and 0.0016, with standard deviations 0.08 and 0.04. Playback highlights exact outcomes rather than collecting random annotations. The normal comparison is an approximation and its displayed bound concerns a standardized CDF. This instrument cannot estimate correlated annotator errors, reveal a latent true preference, or compensate for excluded difficult pairs.