Engineering / research
Learn from preferences with explicit assumptions
Trace preference annotation, reward modeling, reference constraints, and optimization while separating learned judgments from guaranteed behavior.
Your route through engineering
Follow the path or begin with a question. Every lesson is open.
Your place, at your pace. Progress is optional and saved only in this browser when available.
Read / explore / reason
Inside this module
Module 01 research
Learn from preferences with explicit assumptions
Trace preference annotation, reward modeling, reference constraints, and optimization while separating learned judgments from guaranteed behavior.
- 01
Define which comparisons can teach the intended behavior
Build preference pairs with explicit prompt, candidate, rubric, and sampling identities while examining missing coverage and dependent observations.
24 min - 02
Report agreement with its eligible population and uncertainty
Distinguish observable label consistency from latent correctness, and preserve ties, abstentions, grouping, and population differences in noise analysis.
24 min - 03
Fit comparison probabilities while keeping reward coordinates identifiable
Derive logistic pair likelihoods, explain additive score ambiguity and comparison-graph limits, and separate a fitted ranking scale from universal utility.
24 min - 04
Evaluate predicted judgments without declaring a utility meter
Separate ordering accuracy, probabilistic calibration, population shift, and downstream usefulness when evaluating a learned reward model.
24 min - 05
Derive a reference-constrained policy on an explicit support
Analyze a finite-support KL-regularized objective, its normalization, and its limits without mistaking an average penalty for a hard behavioral guarantee.
24 min - 06
Derive the reference-relative classification margin
Connect the finite KL optimum to a Bradley–Terry likelihood, keeping prompt normalization, beta convention, and sequence log-probability semantics explicit.
24 min - 07
Cache the exact reference probability question
Bind precomputed completion scores to model, tokenizer, context, masks, and numerical policy so a fast reference path preserves the intended preference objective.
24 min - 08
Control presentation and length without erasing real task differences
Design paired display-order checks and explicit length analyses while separating observed nuisance effects from genuine quality and causal claims.
24 min - 09
Look for proxy improvement that independent evidence does not support
Track optimization pressure, distribution shift, and held-out judgments without treating a rising learned reward or a synthetic gold model as guaranteed quality.
24 min - 10
Release a versioned behavior change with reviewable evidence
Combine held-out preference results, task requirements, uncertainty, configuration identity, and a tested rollback target into a scoped release decision.
24 min