Reward transformations: stabilize scales without hiding the objective

A fixed positive scale multiplies a pure expected-return objective and preserves its policy ordering when the underlying task, discount, initial distribution, and evaluation convention remain unchanged. It can still change optimization dynamics, gradient clipping, and the relative influence of unchanged entropy, KL, or auxiliary coefficients. An online normalization statistic is not simply one fixed scale: its state depends on data and time. Version and checkpoint that state, define whether evaluation freezes it, and log raw rewards alongside transformed training signals.

Adding a constant per step can favor different episode durations, because a longer trajectory accumulates the offset more times. With a fixed policy-independent horizon and the same discount convention, the offset contribution can instead be a policy-independent constant; the condition matters. Nonlinear clipping or task-dependent scales can change rankings and mixture weights. Do not describe every reward wrapper as harmless numerical normalization. Derive the transformed return for at least one short and one long trajectory before interpreting a learning curve in the new units.

Potential-based shaping adds a term derived from the same discount and a potential difference between successive task states. Its discounted sum telescopes, exposing both the initial potential and a finite endpoint term. Under appropriate conditions, such as a fixed initial distribution and zero potential at task terminal states, that endpoint contribution does not change policy ordering. For infinite discounted tasks, bounded potentials and discount strictly below one give another familiar vanishing-tail argument. State which condition is being used rather than quote policy invariance without its boundary assumptions.

External truncation is especially easy to mishandle: it is not automatically a task terminal state where a potential can be discarded. A bootstrap target and shaping convention must account for the continuing tail consistently. Shaping on reset observations can add an unintended transition between episodes. Partial observations, learned potentials, and changing reward models require further analysis rather than a blanket transfer of the state-based fixed-potential result. Keep the potential version and task boundary contract with the experiment so evaluation can reconstruct which objective was optimized.

A simulated assistant rewarded for tool-call count may learn unnecessary calls, while a robot rewarded only for forward motion may exploit a simulator artifact. Those are objective-specification failures rather than evidence that the optimizer misunderstood the scalar reward. Separate raw task outcomes, constraints, and independent audit metrics from the training proxy. Evaluate adversarial cases and distribution shifts where the proxy and intended goal can diverge. Reward-model agreement or a larger transformed return is not sufficient evidence of useful or safe behavior.

The function lab illustrates only a fixed coefficient scale. Reset, explicitly set coefficient one and position two, predict y, then Play or Step. Compare a fresh Reset with coefficient 0.5 and position two explicitly set; y changes from four to two. This scales a fixed parabola and has no episodes, reward offsets, changing normalization state, shaping potential, or proxy optimization. Use the return calculations and independent task evaluation to test those distinctions rather than treat an unchanged-looking curve shape as policy invariance.