Evaluate predicted judgments without declaring a utility meter

Pair accuracy records whether the model favors the labeled candidate under a declared tie rule. It discards much of the confidence information in the score gap. Log loss and Brier loss assess probabilistic predictions and penalize different aspects of error; a confident wrong prediction can matter even when its argmax is unchanged. Use the same held-out examples and outcome convention when comparing them. A reward model's raw scalar output is not itself a probability: a link such as the specified Bradley–Terry comparison model maps a within-prompt gap to a predicted judgment probability.

Proper scoring rules have an expected-loss interpretation under the actual outcome distribution. Minimizing a finite empirical score does not prove that a flexible learned model recovered that distribution, and model selection on the same examples can overfit the score. Preserve a separate calibration or development set and a final assessment set. Report enough labels, denominators, and probability bins or smooth summaries to make confidence behavior visible. Coarse calibration bins can hide subgroup errors, while small bins can produce noisy apparent deviations. Avoid presenting one aggregate calibration number as a complete model-quality certificate.

A held-out set can ask whether pairs assigned probability near 0.8 are chosen in the predicted direction roughly eighty percent of the time. The answer is conditional on the prompt source, candidate generators, rubric, annotator population, and inclusion rule. A score rescaling or calibration mapping learned there is another fitted artifact requiring version identity and held-out validation. Additive reward offsets do not change Bradley–Terry probabilities, but scaling gaps changes their confidence. None of these operations identifies absolute utility or converts agreement with a rubric into a guarantee of desirable behavior.

Generalization requires additional comparisons that stress the intended shift: unseen prompt families, other response generators, different languages, or failure modes that optimization discovers. Splitting random pairs from the same prompts can understate that challenge. RewardBench 2 constructs multi-skill evaluation evidence and studies downstream relationships, but a benchmark result remains evidence on that benchmark and protocol. Calibration on one population can fail under another, and high pair accuracy may not predict the best reward for a downstream optimization budget. Record where the model was tested and which conclusions remain unmeasured.

Group dependent pairs by prompt or source when designing uncertainty analysis, and keep annotator variation visible. Evaluate score behavior on current-policy outputs rather than only the older generators used for training. For best-of-N selection, inspect how errors among high-scoring candidates change with N; for policy training, compare independent held-out judgments as optimization proceeds. Maintain correctness or task checks alongside preference predictions when those are release requirements. A calibrated predictor of one annotation event can still be an inadequate optimization proxy if relevant behavior lies outside its observation and generalization boundary.

The network lab isolates confidence changes while preserving one decision direction. For each separate view, Reset, set both input x=1.5 and weight w, predict z=wx−0.5, then inspect. At w=1 the output is about 0.7311; at w=2 it is about 0.9241. Both exceed 0.5, but they express different probabilities in this toy unit. The instrument has no labels, calibration fit, or generalization test. It demonstrates why identical argmax decisions do not imply identical probability quality; actual scoring needs held-out outcomes from the declared evaluation population.