Policy promotion: bind the artifact to reproducible evidence
Freeze evaluation semantics, preserve uncertainty and constraint results, and promote an immutable checkpoint with the preprocessing state that actually produced its behavior.
On this page
The question for today
What you’ll understand
- Define an immutable policy and evaluation manifest.
- Calculate paired checkpoint differences without hiding failed constraints.
- Separate finite acceptance evidence from universal performance or reward guarantees.
A policy checkpoint is more than a weight file, and a promotion decision is more than the largest observed return. Observation normalizers, action transformations, environment wrappers, recurrent-state handling, reward conventions, and evaluation sampling can all change the behavior attributed to the same weights. Bind the candidate to an immutable manifest and an explicit evaluation protocol. Original RL evaluation research documents why few-run comparisons and selection can mislead; reproducible promotion should retain uncertainty, failures, and intended-task checks instead of treating one attractive mean as a universal guarantee.
Freeze the candidate and the evaluation contract
Identify weights, architecture, preprocessing and normalization snapshots, action distributions, masks, recurrent initialization, simulator version, wrapper configuration, task horizon, reward reporting, and dependencies. Distinguish a deterministic evaluation action rule from the stochastic training policy; changing that rule changes the evaluated object. A training-resume checkpoint also needs optimizer, RNG, replay, and input state according to its recovery claim, while an inference artifact may contain a smaller deliberately defined subset. Label those artifact types so evaluation does not silently depend on mutable training-process state.
Freeze a suite and seed protocol before selecting the winner. Include intended task outcomes and relevant constraint or failure metrics in addition to the optimized proxy. Keep tuning data separate from held-out acceptance data, and account for repeated selection across checkpoints or hyperparameters. A fixed evaluation seed list supports reproducibility, but repeatedly choosing against that list can overfit it. Version the task suite and preserve the full run manifest so a later rerun does not accidentally use a revised simulator or a normalizer still updating online.
Preserve uncertainty and non-return outcomes
Report per-task and per-run results, failed runs, episode lengths, constraint violations, and resource or deadline outcomes where relevant. Separate independent training replicates from multiple evaluation episodes of one checkpoint. Statistical intervals or bootstrap procedures must respect the actual grouping and dependence rather than resample individual correlated timesteps as independent evidence. A suite aggregate can hide a critical task regression; specify whether acceptance requires a minimum task result or zero observed violations in a defined fixture. Such finite checks still do not prove zero underlying risk.
A promotion gate is a product or research decision rule, not a theorem supplied by PPO or replay correction. It can require protocol identity, a predeclared performance criterion with suitable uncertainty, and separate constraint evidence. An observed improvement does not excuse a schema mismatch or a forbidden action. Keep the evidence attached to the exact artifact digest and record which criterion failed. If evidence is inconclusive, retain the incumbent or collect the specifically needed independent evaluation rather than silently choose the highest point estimate.
Publish one artifact-evidence relationship and test rollback
Write candidate artifacts and evaluation records immutably, validate completeness, then update the deployment reference through the supported atomic promotion mechanism. Concurrent evaluators must not overwrite a newer accepted decision using stale evidence. Retain the incumbent artifact, manifest, and rollback path. A resumed trainer can continue producing new checkpoints without mutating the candidate already under evaluation. After promotion, verify deployed preprocessing and action semantics against the accepted artifact; distribution shift and operational failures require monitoring rather than an assumption that offline reward guarantees future behavior.
The sampling lab illustrates uncertainty only for a hypothetical independent binary success metric. Reset, explicitly set p=0.5 and sample size 20, predict the sample-mean variance, then Play or Step. Compare a fresh Reset with p=0.5 and sample size 80 explicitly set. Variances are 0.0125 and 0.003125. Playback highlights exact possible outcomes rather than drawing evaluations, and its normal comparison is approximate. Real returns need not be Bernoulli or independent, so this instrument does not compute a valid confidence interval for an arbitrary RL benchmark or decide promotion.
Work it through
An improved mean does not override a failed gate
Five paired fixture returns for an incumbent are [10,11,8,9,12] and for a candidate [12,10,11,9,13]. An explicitly limited illustrative gate requires mean paired improvement at least 0.5 and zero observed forbidden actions in this fixture. The candidate has one forbidden action.
Paired differences are [2,−1,3,0,1], summing to five. Mean difference is one return unit. Incumbent mean is ten and candidate mean eleven under this fixture protocol.
The point-estimate criterion one≥0.5 passes. The zero-observed-forbidden-action criterion fails because one violation occurred. The conjunction therefore rejects this candidate despite its improved mean.
This deliberately small arithmetic gate does not supply statistical confidence or establish future risk. A real predeclared evaluation must address task coverage, independent runs, uncertainty, and selection effects, while retaining this concrete violation and its artifact provenance.
The candidate fails the stated gate. Its mean improvement and its constraint failure are both retained rather than collapsed into a single celebratory score.
The living notebook
Explore the model
Interactive lab · sampling
A binary sampling distribution is not an RL acceptance decision
Use independent Reset → explicitly set BOTH p and n → predict → Play/Step runs for (.5,20) and (.5,80). Exact sample-mean variances are .0125 and .003125; playback highlights outcomes, not real evaluations. The normal curve is approximate, and general return data may violate this binary independent model.
- Mean sample proportion
- 0.5
- Sample-mean variance
- 0.0125
- Uniform CDF error bound
- ≤ 0.1062
Exact binomial masses for the mean of n independent Bernoulli trials, compared with normal mass between adjacent half-count boundaries; normal tails outside the finite support remain outside the bars. Playback highlights outcomes, not random samples. The Berry–Esseen bound applies to standardized CDFs, not individual bars or continuity-corrected errors. At p = 0 or 1, normal comparison and that bound are disabled. Normal-CDF evaluation uses a numerical approximation with absolute error below 7.5×10⁻⁸ (at most 1.5×10⁻⁷ per interval, before rounding). Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.
Look forMore samples narrow uncertainty under assumptions; artifact identity, coverage, constraints, and selection remain separate acceptance obligations.
Pause & reason
Check your understanding
Your turn
Try it yourself
Paired differences on three fixed cases are [2,−2,3]. A deliberately limited fixture gate requires mean improvement at least one and zero observed violations. There are no violations and the manifest is valid. Compute the gate result and state what it does not prove.
Take a moment to make a prediction or work it out on paper. Open a hint when you need one.
Give me a hint
Evaluate each stated finite criterion, then keep arithmetic acceptance separate from broader statistical and deployment claims.
Compare with the solution
The mean difference is (2−2+3)/3=1, so the point-estimate threshold passes exactly. With a valid manifest and no observed fixture violations, the stated finite gate accepts. Three cases do not by themselves establish statistical confidence, independence, unseen-task performance, or zero future violations. A broader production or research claim needs the predeclared evaluation population, uncertainty and selection treatment, task coverage, and operational checks. Preserve these results with the exact artifact rather than transfer acceptance to a later mutable checkpoint.
Keep the idea
Take it with you
- Bind evaluation to the complete immutable policy and preprocessing artifact.
- Retain uncertainty, task regressions, and constraint failures alongside return.
- Promote and roll back through a traceable artifact-evidence relationship.
Read further
Original sources and deeper explanations to follow at your own pace.
Optional progress, saved in this browser when available. Revisit any lesson at any time.
Lesson not marked complete.