Preference optimization and alignment

Supervised fine-tuning learns from example responses to prompts. A preference dataset instead presents alternatives and records which response a labeler favors under a rubric. One RLHF approach fits a reward model to those comparisons, then optimizes a policy while penalizing departure from a reference. InstructGPT provides an influential example of demonstrations followed by human preference feedback. Its empirical improvements are evidence for a particular training and evaluation setup, not proof that every preference-trained policy is truthful or universally aligned.

Write the annotation instructions and record ambiguous or disputed comparisons. Preferences can reflect conflicting judgments about style, length, facts, or values. If longer answers are often preferred, a model may learn verbosity instead of stronger reasoning. Collect representative prompts, separate related prompts across splits, and inspect annotator disagreement. Label model-selected or rule-selected pairs with their provenance rather than calling them independent human judgments.

Direct Preference Optimization derives a pairwise classification objective from a regularized preference formulation. For each prompt, compare the policy's likelihood ratio to a fixed reference for the preferred response and for the rejected response. The loss encourages a higher relative score for the preferred response. This can avoid fitting and then sampling against a separate reward model during the DPO update, but it still requires appropriate preference data, optimization, and evaluation. It is not merely supervised training on the preferred text alone.

Use sums of token log probabilities for sequence scores, with consistent tokenization, prompt masking, and end-token treatment. Padding must not contribute to the score. Standard DPO uses sequence log likelihoods; replacing them with length-normalized scores changes the objective and should be named explicitly. Beta appears both in the derivation's regularization tradeoff and in the scaled pair logit. Its practical effect depends on optimization and data, so do not infer a universal update size from its numerical value alone.

Held-out pair accuracy measures whether the model orders similar response pairs as desired. It does not fully test interactive task success, truthfulness, robustness, or the appropriateness of refusal. Evaluate factual and procedural correctness with independent criteria, and compare responses with length and formatting controlled where relevant. Include distribution shifts, conflicting instructions, and cases requiring uncertainty. A policy can improve preference scores by sounding confident while becoming less honest about missing evidence.

Track capability regressions as well as preferred behavior. Evaluate the base or supervised reference under the same prompts and decoding settings, report important slices, and document data filtering. Preserve the training recipe, reference checkpoint, preference dataset identity, and scoring convention. Human feedback and automated judges can both carry systematic bias. Treat alignment as measured behavioral requirements with stated limits, rather than a binary property bestowed by one loss.