Behavior versions: a trajectory’s probabilities belong to its actor

Attach an immutable behavior-policy identifier to each rollout or, if the actor can change policy inside it, to each transition. Store the action log probability under the actual behavior distribution, not a probability recomputed later from the learner’s current model. Exploration mixtures, temperature, action masks, action transformations, recurrent context, and observation preprocessing can all change that distribution. For continuous actions, the density must correspond to the actual represented action and transformations. A weight-file digest alone is incomplete if the actor applied a different normalization snapshot or sampling rule.

Define when actors adopt a new snapshot and ensure they do not read half-updated parameter arrays. Freezing behavior within an unroll simplifies provenance, though another design can log per-step versions and contexts. Track environment version, episode identity, model state, and probability metadata together through transport and replay. If an actor reports an impossible zero probability for an action it supposedly sampled, investigate the record rather than stabilize the denominator with an arbitrary epsilon and declare the mismatch corrected.

Subtracting behavior version from learner version gives an update-count lag. The same count can correspond to tiny or enormous probability changes, depending on learning rate, data, architecture, and update behavior. Wall-clock age is another distinct quantity and depends on transport and clock assumptions. Monitor behavior-age distributions, ratios, and relevant divergence diagnostics rather than impose a universal acceptable number of versions. Slow actors can contribute a systematically different task or state distribution, so indiscriminately dropping their rollouts can also change the dataset.

At one fixed history, exact importance weighting can transform a behavior expectation to a target expectation when support and integrability hold. Full trajectory corrections can have large variance, and an action-level identity is not a blanket guarantee for bootstrapped learning. Clipping weights controls magnitude but changes the estimator. IMPALA’s V-trace uses specified truncated ratios and has a correspondingly scoped target and analysis; it is not simply “multiply everything by a capped ratio and get an unbiased on-policy algorithm.” State the chosen estimator and its assumptions explicitly.

An on-policy pipeline can stop admitting old-version rollouts at a batch boundary, while an off-policy learner can retain data with a justified correction and sampling scheme. Both designs need explicit policies for delayed packets, retries, worker restarts, and recurrent sequences. Preserve discarded-data counts and task mix. A queue that is operationally healthy can still deliver data outside the algorithm’s acceptable provenance. Validate the behavior probabilities using a saved actor snapshot on a small replayable fixture before trusting aggregate learning curves.

Use the probability lab for a binary action-frequency analogy. Reset, explicitly set p=0.25 and n=20, predict the mean count, then Play or Step. Compare a fresh Reset with p=0.5 and n=20 explicitly set. The means change from five to ten even though the action-count format is identical. These are known independent Bernoulli distributions, not actor traces, policy updates, importance estimates, or V-trace. Playback does not sample trajectories or infer behavior probabilities from a version number.