Monitor with a rule whose error guarantee survives stopping

At one prespecified valid analysis, a level-α test bounds its false-positive probability under the null. Testing the same accumulating experiment at many times creates multiple chances to cross the threshold. Those tests are usually dependent, so an independence formula is not a universal description of peeking. The important point is that marginal validity at each time does not imply that the probability of any rejection over the run stays below α.

One simple conservative solution declares a finite set of L look times and assigns each look a portion of the total error budget. If every per-look p-value is valid under the null at its declared analysis time, the union bound controls the probability of any false rejection without requiring the looks to be independent. Arbitrarily adding more looks after seeing an almost-significant result is outside that declared guarantee unless the design already reserves and allocates the necessary error.

A confidence sequence is built to cover the target parameter simultaneously across the allowed observation times with a stated probability. Its construction must match the data process: boundedness or moment conditions, randomization or sampling assumptions, dependence, and any adaptive allocation. A fixed-sample confidence interval printed at every timestamp is not a confidence sequence merely because the chart is animated. Use a validated construction rather than changing the label on an ordinary interval.

A valid sequential method can support a stopping rule based on its bounds, such as stopping for benefit when the lower bound exceeds a practical threshold. It still needs an appropriate causal estimand and complete or modeled outcomes. Delayed feedback, treatment-dependent missingness, and a changing user population are not repaired automatically by optional-stopping validity. Multiple metrics, model variants, and repeated experiment restarts also need an error-control plan if the decision claims rely on them.

An experiment may have a safety stop for severe latency, error, or other operational regressions even when it is not ready to claim benefit. Record such a stop and its reason, and do not present the truncated result as though the originally planned fixed-horizon analysis completed unchanged. Likewise, a futility rule can save resources without proving the effect is exactly zero. The interpretation depends on the rule and the available uncertainty.

Document what counts as a look, when outcomes are mature enough to enter, what happens when a data-quality alarm fires, and how a stopped experiment is reported. Preserve the entire trajectory and the procedure version, not only the most favorable timestamp. A good monitoring system makes it easy to protect users while being honest about which statistical conclusions the stopped data support.