Human disagreement is evidence to investigate, not a truth label to erase

Define outcome criteria, allowed alternatives, process constraints, severity categories, and treatment of incomplete evidence. Provide reviewers the artifacts needed to assess those criteria, including authoritative effects where relevant, while avoiding unnecessary identity cues about the system being compared. Blinding can reduce some expectations or preferences, but removing all trace context may make the task unjudgeable. Randomize presentation order where order should not matter and record the actual review interface. An attractive explanation or longer answer should not substitute for evidence of the requested state change.

Collect independent initial labels before showing another reviewer's decision. Preserve uncertainty or an unresolved category when the protocol permits it instead of forcing false precision. The judge-evaluation research of Zheng and colleagues examines how assessment systems can respond to presentation and other biases; a human review workflow also needs its own direct audit rather than assuming that a named bias correction eliminates every error. Sample difficult and apparently easy cases, and inspect whether disagreements cluster by task family, language, evidence quality, or ambiguous rubric wording.

Cohen's kappa compares observed agreement with the agreement expected from the two reviewers' marginal label proportions under its chance baseline. A large majority class can produce high raw agreement, and changing prevalence can change kappa even with a similar number of disagreements. Report the underlying table and marginal rates rather than treating one threshold as a universal approval rule. A high kappa can coexist with a shared misconception or missing evidence because both reviewers can make the same mistake. The statistic is a description of labels under a protocol, not a certificate of truth.

Review sampling also determines what the statistic represents. A deliberately oversampled set of disputed or high-risk cases does not automatically estimate agreement across all ordinary tasks without appropriate weighting. Repeated traces from one task can cluster, and uncertainty analysis should respect that unit. Keep independent review labels separate from the later adjudicated reference labels. Once reviewers discuss a case, the resulting consensus can be useful but no longer measures their original independent agreement. Record which evidence resolved the issue rather than simply replacing both earlier labels with the final answer.

Use a documented adjudication process for disagreements, such as another reviewer examining the task contract and decisive state evidence. Majority vote can be a rule, but it is not automatically ground truth. Distinguish an annotation mistake from a legitimate alternative interpretation, grader bug, or incomplete task specification. If adjudication changes the rubric, version it and reconsider affected earlier cases rather than applying the new interpretation only where it benefits one system. Preserve original labels, reasons, final labels, and unresolved cases with appropriate access control for sensitive trace contents.

The probability lab models only a stipulated independent disagreement count. Reset, explicitly set BOTH p=0.2 and n=20, predict four disagreements on average, then Play or Step. Reset and re-set BOTH p=0.1 and n=20; predict two. Edits retain t; playback highlights exact binomial outcomes. The toy has no reviewers, rubric, marginal table, kappa calculation, or adjudication. It cannot determine whether disagreements reveal reviewer error or task ambiguity; those interpretations require the evidence and review process defined above.