Engineering / research
Infrastructure for policy improvement
Follow trajectories, rewards, policy versions, replay, and distributed updates through reproducible reinforcement learning systems.
Your route through engineering
Follow the path or begin with a question. Every lesson is open.
Your place, at your pace. Progress is optional and saved only in this browser when available.
Read / explore / reason
Inside this module
Module 01 research
Infrastructure for policy improvement
Follow trajectories, rewards, policy versions, replay, and distributed updates through reproducible reinforcement learning systems.
- 01
Environment transitions: preserve the boundary that the learner needs
Define observation, action, reward, termination, and truncation records before deriving bootstrap targets or resetting an episode.
29 min - 02
Vector execution: autoreset modes change which rows are transitions
Trace pinned Gymnasium vector reset semantics, final observations, reset-only slots, and per-environment recurrent state.
29 min - 03
Behavior versions: a trajectory’s probabilities belong to its actor
Track policy and preprocessing identity, derive a conditional importance ratio, and state support and clipping limits before using stale rollouts.
29 min - 04
Returns and GAE: finite fragments need explicit tail and trace masks
Derive discounted tails and generalized advantage recursion while keeping terminal, truncated, and collector boundaries distinct.
29 min - 05
Replay priorities: correct the actual sampling distribution
Derive proportional sampling and importance weights while preserving transition identity, support, and the limited meaning of fixed-buffer correction.
29 min - 06
On-policy batches: freeze behavior identity across PPO reuse
Define rollout and update phases, preserve old probabilities, and distinguish PPO’s clipped surrogate from a hard policy-distance guarantee.
29 min - 07
Actor-learner transport: throughput needs trajectory identity
Bound queues and policy age while preserving sequence, retry, and actor-incarnation semantics across distributed rollout transport.
29 min - 08
Reward transformations: stabilize scales without hiding the objective
Separate fixed scaling, duration-dependent offsets, potential shaping, and proxy failures while preserving raw evaluation signals.
29 min - 09
Seeds control streams, not every source of nondeterminism
Inventory random generators, logical actor identity, execution order, and evaluation variability without promising cross-platform bitwise replay.
29 min - 10
Policy promotion: bind the artifact to reproducible evidence
Freeze evaluation semantics, preserve uncertainty and constraint results, and promote an immutable checkpoint with the preprocessing state that actually produced its behavior.
29 min