Derive a reference-constrained policy on an explicit support

Fix one prompt and a finite response set Y. Let the reference q assign strictly positive probability to every response in that set, let rewards r(y) be finite real values, and choose β>0. Optimize over normalized policies p on Y. The reward expectation is under the candidate policy, and D_KL(p||q) averages its log density ratio to the reference under that same candidate policy. These choices matter: reversing the KL or changing the sampling distribution defines another objective. Use natural logs and state the reward scale so the coefficient's numerical meaning is reproducible.

A constrained formulation can require an average KL no greater than a budget κ; its Lagrangian introduces a multiplier. Merely setting a positive penalty coefficient in an approximate optimizer does not certify that a chosen hard budget was met. Measure the resulting divergence under the declared distribution and estimator, including uncertainty. Average KL over prompts is also different from a per-prompt bound and from a per-response probability-ratio bound. Keep those scopes distinct when a training report says a model stayed close to its reference.

The optimum weights each reference probability by exp(r/β) and divides by the resulting positive finite partition function. One way to establish this is to rewrite the objective as β log Z minus β times KL to that normalized distribution. Nonnegativity of KL makes the maximizing policy explicit. The result is unique on the stated simplex because the reference is positive and rewards finite. Adding a prompt-only reward constant changes Z and the objective value but cancels from the optimal policy; this matches the score-origin ambiguity of within-prompt preference data.

Support cannot be ignored. If the reference assigns zero probability to a response, assigning it positive candidate mass makes D_KL(p||q) infinite under the usual convention. One can instead define the finite problem only on the reference support, but then unsupported responses are outside the allowed optimization domain. Infinite sequence spaces require further conditions ensuring a finite normalizer and well-defined expectations; the finite derivation alone does not supply them. A neural policy trained with finite data may also fail to represent or find the ideal optimum, so the formula is a conditional mathematical result rather than a training-success guarantee.

For a fixed reward function, increasing this β weakens the exponential reward tilt and moves the ideal distribution toward the reference. Rescaling rewards while holding β fixed changes the problem; matching the reward scale therefore matters as much as reporting β. In preference-derived objectives, β can also appear inside a classification margin, so a local loss response should not be confused with the entire fitted-policy tradeoff. Evaluate current outputs, measured divergence, diversity, and task outcomes together. A close distribution can still preserve the reference's failures, and a high reward can still reflect a learned proxy error.

The probability lab can display the finite two-outcome distributions used below. For each separate view, Reset, set both n=1 and p to 0.5 or 0.75, predict the two bars, then inspect. A one-trial binomial is exactly a Bernoulli distribution, so the bars visualize the reference and ideal toy-policy masses. The instrument does not optimize rewards or compute KL; calculate those from the stated distributions. It has no language model or behavioral verification, and the two outcomes are a deliberately finite mathematical domain, not a realistic response space.