Look for proxy improvement that independent evidence does not support

A learned reward is fitted to a finite comparison process. Optimization deliberately searches for outputs with high scores, which can concentrate errors that were rare in an ordinary held-out sample. More best-of-N candidates, more policy updates, or weaker regularization can increase that selection pressure in different ways. Report those axes and the current response distribution rather than compare reward means across incompatible checkpoints or reward versions. An unchanged frozen reward model provides a consistent proxy scale for the diagnostic, while raw scores from separately normalized models may not be directly comparable.

Independent evaluation can include held-out human comparisons, task verification, rubric-specific checks, and another model whose failure modes are examined. Gao and colleagues study overoptimization in a synthetic setup where a larger fixed reward model supplies a gold-standard proxy for humans. Their gold score is a defined experimental target, not literal human utility or a guarantee about every deployment. Preserve that distinction when using the paper's results. A second judge sharing training data or style biases with the optimized model may provide less independence than its separate name suggests.

Evaluate a predefined set of optimization checkpoints on the same controlled prompt distribution, with blinded candidate identity and balanced presentation. Track proxy reward, independent win or task scores, measured divergence, response length, refusals, and other relevant slices. A reward rise accompanied by a decline in held-out evidence is a diagnostic pattern; identify concrete response changes before attributing the cause. The pattern can also reflect evaluation noise, rubric changes, or population drift, so uncertainty and protocol identity matter. Avoid using the same repeatedly tuned development score as untouched confirmation evidence.

A simple scalar counterexample makes the mechanism explicit. Suppose the independent quality diagnostic peaks at z=1 while the proxy peaks at z=0. Moving from z=2 to 1 improves both, but moving farther toward zero keeps improving the proxy while worsening the diagnostic. This proves a possibility under the specified functions, not inevitability for every learned reward or optimization algorithm. Real systems need examples and held-out measurements to determine whether a comparable mismatch exists, which behaviors drive it, and whether additional data or a changed objective addresses that mismatch.

Possible responses include selecting an earlier evaluated checkpoint, limiting search or updates, revisiting comparison coverage, and collecting independently checked examples from the newly exposed response region. A KL penalty constrains a distributional tradeoff but does not make a misspecified reward true. Reassess the evaluation protocol after changing it, keeping fresh confirmation evidence where repeated selection has occurred. Preserve reproducible artifacts and a known approved rollback target so a release decision does not depend on reconstructing an old checkpoint from a mutable name.

The optimization lab implements gradient descent on z²/2. Reset, set both step size η=0.5 and initial z₀=2, predict the first two updates, then Step twice. The model uses z_k=z₀(1−η)^k, giving 1 and 0.5. Proxy rewards are therefore−0.5 and−0.125, while the separately calculated q values are 0 and−0.25. The instrument plots its quadratic loss, not q, annotations, language behavior, or a reward-learning process. Calculate q explicitly to see how one optimized quantity can improve while another stated diagnostic deteriorates.