On-policy batches: freeze behavior identity across PPO reuse

Assign a batch identifier and freeze the behavior snapshot, action-distribution settings, preprocessing state, and old action log probabilities that define it. In the scoped pipeline here, advantages are computed under the chosen old critic and boundary rules, then held fixed for the batch’s surrogate updates. Another algorithm may recompute them, but that difference must be explicit. A fixed-length fragment can end before the task episode ends and use an appropriate bootstrap value; on-policy batching does not require inventing a task termination whenever the collector reaches its target length.

Actors that have not adopted the requested snapshot can produce a mixture of behavior versions. Decide whether to wait, reject, or use a different justified algorithm, and retain the resulting task and actor coverage. Dropping every slow actor can bias the batch toward easier or faster environments. Count valid transitions rather than reset-only vector outputs, and preserve recurrent sequence boundaries when shuffling minibatches. A uniform array of old log probabilities is not enough if its entries were computed using the wrong masks or a different behavior distribution.

The clipped PPO term takes the minimum of an ordinary ratio-weighted advantage and a clipped-ratio version. For positive advantage it stops rewarding some further increases beyond the upper clip. For negative advantage, the relevant pessimistic branch can differ. This is not equivalent to replacing every ratio in every expression by its clipped value. Work through both signs before implementing the objective, since a sign error can reward exactly the policy changes the surrogate was intended to discourage.

The objective does not enforce that all actual action ratios stay inside the clip interval, nor does it provide a universal hard bound on KL divergence or monotonic improvement. Shared parameters couple actions and states, minibatch steps interact, and the critic and entropy terms add further forces. Monitor the relevant diagnostics and define any early-stop rule as part of the implementation. Multiple PPO epochs reuse the same old-policy batch under this surrogate; they do not erase behavior provenance or automatically justify arbitrary older off-policy trajectories.

Log the rollout batch, behavior digest, learner update, epoch number, and valid-item count at every update. Verify that old log probabilities do not silently change when the current model changes. A ratio of one after every epoch may indicate that old probabilities were recomputed from the current policy, making the intended reference disappear. Test positive and negative advantages, masks, recurrent sequences, and partial minibatches with a hand-derived fixture. Compare objective components before looking for an improved episode-return curve.

The network lab illustrates one changing binary-action probability. Reset, explicitly set input one and weight one, predict the sigmoid output, then Play or Step. Compare a fresh Reset with input one and weight zero explicitly set. Its fixed bias minus 0.5 gives outputs about 0.62246 and 0.37754. This single unit can stand for a hypothetical Bernoulli action probability, but the instrument does not preserve an old policy, calculate a PPO objective, collect trajectories, or update parameters. The clipped-surrogate arithmetic must be checked separately.