Replay priorities: correct the actual sampling distribution

A replay item needs an immutable logical identity, behavior provenance, episode and timestep links, boundary masks, and any recurrent or multistep context required by the learner. Capacity management may store these in a ring, but a physical slot is not a permanent item identity. Give each slot reuse a generation or equivalent identity so a delayed priority update cannot modify a new item that replaced the sampled one. Record the sampling probability and population size associated with the draw, because priorities and the population can change before the learner returns its update.

In proportional prioritized replay, a positive priority raised to an exponent determines relative sampling mass. A small positive priority floor can preserve support for low-error items, but the actual probability must include every implemented mixture, partition, or sequence-sampling rule. A two-stage distributed sampler needs the product or otherwise derived final probability, not just its local leaf weight. Sequence priorities, duplicate handling, and sampling without replacement require their own probability model. State the population being corrected: uniform stored transitions is different from uniform episodes or the current policy’s state distribution.

At beta one, unnormalized weights can recover the uniform fixed-buffer expectation of a specified contribution when sampling probabilities are correct and positive. Beta below one leaves a different expectation. Large inverse probabilities can increase variance, motivating partial correction or other controls, but those choices change the estimator. A common deterministic normalization by the maximum buffer weight rescales the whole expected update rather than preserving its original magnitude; a minibatch-dependent normalization can add further bias. Do not call all normalized implementations exactly unbiased merely because their exponent reaches one.

This algebra concerns replay selection, not all sources of RL error. The stored data may come from old policies, TD targets may use approximations, and bootstrap learning can couple the contribution to the parameters and target network. A fixed-buffer identity does not supply missing action support or repair an incorrect terminal mask. Inspect priority distributions, effective sampling probabilities, item age, duplicate rate, and task mixture. A large TD error can reflect an informative surprise, noise, or corrupted data; repeatedly replaying it does not automatically improve the intended task.

Use a tiny buffer with distinguishable values to check the sampler and expected weighted contribution independently from a neural-network learner. Test overwrite races, duplicate draws, priority-update retries, and sequence boundaries. If a learner updates priorities after an item has been evicted, reject or safely ignore the stale identity rather than assign the error to the new occupant. Track admission and eviction policy as well as priorities: the population itself determines which experiences can ever be sampled. A sampler cannot recover a missing task distribution solely by weighting the items that remain.

The probability lab illustrates one item’s hypothetical draw frequency. Reset, explicitly set p=0.25 and n=20, predict the expected count, then Play or Step. Compare a fresh Reset with p=0.5 and n=20 explicitly set; means are five and ten. These known independent binary trials do not implement a replay tree, changing priorities, slot generations, or importance weighting, and playback does not draw random minibatches. Use the actual fixed-buffer arithmetic to validate correction rather than infer it from the animated bars.