Fit comparison probabilities while keeping reward coordinates identifiable

For a prompt x and two candidate responses, the Bradley–Terry construction assigns positive strengths proportional to exponentiated scores. The probability of choosing A is exp(r_A)/(exp(r_A)+exp(r_B)), equivalently sigmoid(r_A−r_B). This is a model of the annotation event under the collection protocol, not a proof that every human has one stable scalar utility. Fix the link scale or temperature convention before interpreting score gaps. Multiplying scores changes probabilities at a fixed temperature, while adding the same constant to both leaves the probability unchanged.

Fit binary outcomes with their log likelihood, keeping label direction and display mapping correct. A preferred response contributes negative log sigmoid of its score advantage. Soft labels or tie models require an explicitly chosen extension; forcing ties into arbitrary binary winners changes the data model. For neural reward models, one scalar head may share parameters across many prompts, but that parameter sharing is a modeling assumption about generalization. It does not create direct observed comparisons between unrelated prompts or grant a universal interpretation to their raw score levels.

Adding a prompt-specific constant c(x) to all response scores preserves every within-prompt difference. A centering constraint or anchor selects coordinates but does not provide new preference evidence. In a free-score comparison graph, disconnected components can have independent offsets, leaving relative levels between components unsupported. A connected graph supplies links but is not enough by itself to guarantee a finite unregularized maximum-likelihood estimate. If one response wins every observed comparison in a separable direction, the likelihood can keep improving as the corresponding gap grows without a finite optimum.

Regularization, priors, bounded parameterizations, and additional comparisons can make practical fitting better behaved, but their effects must be reported. A finite optimizer output after a fixed number of steps is not proof that the unconstrained statistical optimum is finite. Examine label conflicts, graph coverage, held-out likelihood, and sensitivity to regularization. A scalar-score model also imposes structure on preference patterns; systematic cycles, subgroup differences, or context omissions can show that one shared scale is inadequate. Treat those observations as model-diagnostic evidence rather than automatically declaring disagreeing annotators wrong.

Evaluate both ranking errors and probabilistic losses on appropriately held-out prompt groups. A large score margin can be confidently wrong under a new response generator or rubric, and an accurate ordering does not guarantee calibrated probabilities. Preserve the model checkpoint, tokenizer, prompt formatting, terminal-token selection, and dataset version used to compute scores. A reward score produced from a truncated answer may describe a different observation from the one annotators judged. Inspect slices and examples rather than compare raw checkpoint score means whose origins or scaling may differ.

The network lab demonstrates one logistic link with a fixed bias. For each separate view, Reset, set both weight w=1 and input x, predict the logit wx−0.5, then inspect. At x=0.5 the logit is zero and output 0.5; at x=1.5 it is one and output about 0.7311. These are abstract score-gap examples, not learned response scores. The instrument has no pair dataset, comparison graph, optimization, or utility measurement. Its fixed bias must be included when mapping the visible controls to a desired logistic probability.