Treat the scoring process as part of the measurement system

Define each dimension separately: factual support, task completion, instruction compliance, clarity, or another stated construct. Give positive, negative, and boundary examples, and specify whether a partially correct answer can receive credit. State which external sources or tools annotators may use and when they should abstain. Training and pilot annotation can reveal ambiguous instructions before the confirmatory evaluation. After a rubric revision, preserve its version and decide whether earlier cases need rescoring; mixing incompatible scoring definitions can create an apparent model change without a model change.

Blind model identity when it is irrelevant to the task, randomize presentation order where appropriate, and avoid exposing other annotators' votes before independent judgments are collected. Collect the context required by the rubric without revealing an intended answer through evaluator instructions. Annotators may need specialized knowledge for technical claims, while preference judgments can depend on the target user population. Document recruitment and qualification so the evaluated construct is clear. A panel drawn from one population is not automatically a representative measurement of another population's preferences.

Raw agreement counts matching labels among jointly labeled cases. Report the number of eligible cases and jointly labeled cases as well, because easy cases may be more likely to receive complete labels. A high agreement rate after dropping unscoreable outputs can hide a low coverage rate. Chance-corrected or ordinal reliability summaries may answer additional questions, but none replaces inspecting which dimensions and slices generate disagreements. Use a declared treatment for ties, invalid responses, and abstentions, and retain the original labels when an adjudicator supplies a final operational decision.

For items with several ratings, the empirical distribution over categories describes the observed panel. It need not equal a probability that a factual statement is correct. Pavlick and Kwiatkowski show in textual inference that disagreement can persist rather than disappear with more judgments, motivating attention to distributions of plausible interpretations. For a support-assistant evaluation, distinguish a factual citation error from a legitimate disagreement about tone. Adjudication can resolve a release workflow, but its policy and evidence should remain visible instead of erasing uncertainty from the dataset.

Reevaluate a fixed calibration set when recruiting a new annotation vendor, changing language instructions, or introducing a new rubric. Look for systematic shifts by annotator, task type, and response style, while protecting the final human holdout from tuning. Repeated ratings from the same person and multiple ratings of the same answer are not independent new deployment examples. Choose uncertainty units that match the inference goal. A genuine improvement in agreement may come from clearer measurement, but it should not be mislabeled as improved assistant quality.

The probability lab isolates an agreement-count assumption. Reset, set both p=0.8 and n=10, predict the expected number of agreements, then inspect. The mean is eight agreements for ten independent cases with stipulated constant agreement probability. The instrument has no annotators, categories, rubric, or truth labels, and its independence assumption may not hold for a real panel. It cannot decide whether agreement is valid or whether abstentions are selective. Use it to separate a modeled count from the substantive measurement process that produces the actual judgments.