Validate automated judges against a protected human measurement process

A judge can return the same verdict repeatedly while consistently rewarding verbosity or a favored response position. Measure repeatability separately from agreement with the declared human reference procedure. That reference also has rubric and annotator uncertainty, so report its construction rather than call it infallible truth. Choose evaluation tasks relevant to the judge's role: factual verification may require source evidence or executable checks, while user preference depends on the specified population. A judge that works on one answer style can fail on a different domain or model family.

Zheng and colleagues document position, verbosity, and self-enhancement effects in their studied judges. Use controlled checks instead of assuming every new judge inherits the same magnitude of bias. Swap response order while mapping decisions back to model identity, blind names, and test semantically matched style variants where justified. Inspect invalid formats, ties, abstentions, and refusals. A swap can measure sensitivity, but agreement after swapping is not itself factual correctness. Any mitigation changes the measurement procedure and should be frozen before the final evaluation.

If a judge supplies numerical confidence, define the event that number is supposed to predict. Confidence in choosing an answer is different from confidence that every claim is correct. Compare probabilities with independent reference outcomes on held-out cases and report uncertainty, support counts, and slices. Calibration bins are design choices and can hide local errors; a global average can look good while one important category is poor. A score such as mean squared probability error combines calibration and discrimination properties rather than isolating every source of judge failure.

Use one set for developing prompts or fitting a calibration mapping and a separate protected set for final judge assessment. Reusing the same human labels to select and validate the judge creates selection leakage. Keep human review for consequential, novel, or low-support cases according to the release policy. A model-generated explanation may help diagnose an error, but fluent reasoning and a high confidence number are not independent evidence. When the evaluated assistant changes response style or domain, recheck the judge rather than assume an old calibration relationship transfers unchanged.

Pin the judge checkpoint or provider revision when possible, prompt, examples, system instructions, temperature, output parser, tie rule, and reference retrieval snapshot. Preserve raw judgments and failures, subject to the evaluation's access restrictions. Parser changes can alter which verdicts count, and a provider alias may change without an explicit local code change. Compare judge error and disagreement by relevant slices using sufficient independent support. Escalate unexplained drift and retain the ability to rescore preserved model outputs under a newly documented measurement version.

The sampling lab describes agreement evidence under one stipulated probability. Reset, set both p=0.7 and n=100, predict the mean and variance, then inspect. The sample-mean distribution has mean 0.7 and variance 0.0021 under independent identical reference-match events. It does not contain a judge, a confidence predictor, a human panel, or a calibration fit. Playback highlights outcomes rather than collecting judgments. The display helps distinguish an assumed match probability from a judge's verbal certainty; estimating that probability in practice requires the protected, independently labeled cases described above.