Decide with uncertainty and retain a compatible rollback state

Define a primary outcome with a minimum worthwhile improvement and guardrails with explicit allowed degradation. Use bounds calibrated for the chosen monitoring, candidate-selection, and multiple-metric procedure. One possible conservative rule requires the primary lower bound to exceed the benefit threshold and every guardrail upper bound to stay below its degradation limit. This is an example decision policy, not the only rational way to trade uncertainty, cost, and user impact.

Failing such a rule is not automatically evidence of harm. A guardrail interval crossing the limit may be inconclusive, while an interval entirely above it supplies stronger evidence of excessive degradation under the analysis assumptions. Likewise, a primary interval containing zero does not prove equivalence. If equivalence or noninferiority is the goal, define the margin and method deliberately. Keep absolute percentage-point changes distinct from relative percentage lift when communicating the practical size.

Review assignment, exposure, SRM, outcome maturity, missingness, and model-version lineage before interpreting the scorecard. An adjusted estimator or sequential procedure cannot repair a different causal population caused by broken logging. Report the analyzed cohort, horizon, and important subgroup uncertainty. An average benefit can conceal a poorly measured or harmed population, but searching many unplanned slices and reporting only the most favorable ones creates another selection problem.

Translate the estimate into the context of the rollout. A result from low saturation may not extrapolate through network effects or shared infrastructure at full adoption. A gradual deployment can test operational limits while maintaining the original inferential boundaries, but new phases and changing allocation should be recorded. Preserve emergency stop criteria, and keep a model-quality conclusion separate from a capacity failure or missing-data incident so a rollback does not erase the reason for the decision.

Give each tested candidate an immutable manifest binding weights, preprocessing, feature definitions, tokenization where relevant, output mappings, thresholds, serving code, and dependent index or configuration versions. Log that manifest identity with assignment and intervention events. A mutable name such as latest is insufficient to reconstruct what produced the observed outcomes. Validate compatibility of the previous manifest before relying on it as a rollback target, including feature schema and input/output behavior.

A rollback changes future serving; it does not undo messages already sent, user learning, cache effects, or delayed outcomes from prior exposure. Preserve those units’ original attribution and follow-up records. Record the selected manifest, traffic/configuration epoch, stop reason, and analysis state, then verify that actual served versions and operational metrics match the intended recovery. Retaining tested bytes and a reproducible decision record makes the next investigation about evidence rather than memory of which model was supposedly running.