Derive the reference-relative classification margin

Use the previous finite response domain with positive normalized reference q, finite rewards, and β>0. The KL-regularized optimum satisfies p*(y)=q(y)exp(r(y)/β)/Z. Taking logs and rearranging expresses reward as β log(p*/q)+β log Z. The normalizer depends on the prompt but not which response is compared within that prompt. Therefore it disappears from a within-prompt reward difference. Comparing unrelated prompts does not generally cancel their different constants, so raw implicit-reward levels should not be interpreted as a universal cross-prompt utility scale.

The change of variables can also associate a positive normalized candidate policy with an implicit reward β log(pθ/q), choosing a particular representative of the additive-equivalence class. That is a mathematical parameterization, not evidence that the policy recovered the reward process behind real annotations. A restricted neural family, finite data, inconsistent preferences, and optimization error all limit that inference. Preserve the reference identity and supported domain. If a comparison response has zero reference probability, its log ratio is not a finite quantity covered by this derivation; arbitrary clipping creates an additional approximation or changed objective.

Assume the unit-temperature Bradley–Terry link for preference labels. Substitute the reward difference to obtain a logistic probability whose margin is β times the chosen response's policy/reference log ratio minus the rejected response's corresponding ratio. The negative log probability of the observed winner is the original sigmoid DPO loss. The reference terms matter: the objective compares relative changes from the reference, not simply whether the policy assigns the winner more absolute probability than the loser. A reference-free variant or alternative preference loss defines a different stated problem.

Keep β's convention explicit. Here it is the positive KL penalty multiplier in the underlying reward objective and also multiplies the log-ratio difference in the DPO margin. If the annotation link has a separate positive temperature τ, the substituted margin scales by β/τ instead. Software or papers using reciprocal-temperature notation need conversion. At a fixed observed log-ratio gap, changing β changes the local classification margin, but the fitted implicit reward also changes during training; that local calculation alone is not a monotonic guarantee about the final model's measured KL or behavior.

For an autoregressive response, the conditional log probability is the sum of its token log probabilities given the prompt and earlier response tokens, under a consistent termination convention. The prompt supplies context but its own likelihood is not added to a completion-only score. Mask padding, align shifted targets, and handle EOS consistently on policy and reference paths. Dividing each sequence sum by its own token count is generally not the same conditional sequence probability and can change pair ordering. If length normalization is desired, name and validate that changed objective rather than claim the original derivation is untouched.

Pinned TRL 0.23.1 distinguishes several loss variants and computes completion log-probability sums for its ordinary sigmoid path; other options can alter this behavior. Inspect the exact configuration and implementation instead of applying every current feature to the original paper. The network lab demonstrates the logistic margin only: for each separate view, Reset, set both w=1 and x to 1 or 1.5, predict z=wx−0.5, then inspect. Outputs are approximately 0.6225 and 0.7311 for margins 0.5 and 1. It has no reference model, tokenization, preference dataset, or optimizer and cannot demonstrate DPO training success.