Environment transitions: preserve the boundary that the learner needs

Record the observation or history used to choose the action, the actual executed action, the resulting reward, and the post-action observation before a new episode replaces it. Include environment identity, episode identity, timestep, action mask where relevant, and preprocessing version. An observation can be partial: a value function of the observation alone then need not represent a Markov state value. Recurrent policies require the appropriate history or hidden-state contract. A tensor’s shape says nothing about whether it belongs to the correct episode or follows the executed action.

Validate action and observation spaces, dtypes, reward conventions, and exceptional outcomes. A simulator timeout, a successful task completion, and an infrastructure failure are not interchangeable rewards. Decide whether a failure produces a valid task transition, an externally cut fragment, or unusable data, and report it separately. Logging privileged simulator state in an information dictionary does not mean the acting policy may consume it. Keep actor-visible inputs distinct from diagnostics so that offline evaluation does not accidentally grant the policy information unavailable during deployment.

Termination means the task’s defined terminal condition has been reached. External truncation stops collection without necessarily ending the underlying decision process, so a continuing-task value target can bootstrap from the true post-action observation. A finite horizon that is part of the task is different: its end is task termination, and remaining time must be represented when needed for the Markov description. Replacing both flags with “done” loses the information required to distinguish those objectives. If both flags are true, the terminal condition still removes continuation under the target defined here.

A bootstrap mask is not permission to connect two episodes. At an external cutoff, the target can use the value of the old episode’s final observation while the next collected observation comes from reset. Recurrent state and multistep traces must respect that episode boundary. Likewise, a fixed-length collector fragment may bootstrap without requiring an environment reset. Define the task, reset boundary, and stored sequence boundary separately, then test them with observations carrying unmistakable episode and timestep identifiers.

Create fixtures for ordinary continuation, terminal success, terminal failure, external time limit, and collector-only fragment cuts. Check the stored final observation and masks directly, not only a noisy episode-return chart. Include a case where reset observations have very different values from final observations, since that makes accidental substitution obvious. A one-step contract error can contaminate every return and advantage afterward. Preserve enough provenance to replay the transition adapter independently from the stochastic optimizer and policy update.

The Markov lab illustrates only a deterministic absorbing-state limit. Reset, explicitly set transition p=1 and q=0, predict the probability of state one after one Step, then Step once. For a separate comparison Reset and explicitly set p=0 and q=0 before predicting and taking one Step. The probabilities are one and zero from the fixed initial state zero. This two-state distribution model has no actions, rewards, Gymnasium flags, or value targets; it cannot validate the collector contract or decide which task condition should terminate.