Calibrate a family of drift tests, including repeated looks

The two-sample Kolmogorov–Smirnov statistic is the largest absolute gap between empirical cumulative distributions. Its ordinary continuous-null calibration assumes independent samples from their respective populations, with the null asserting identical continuous distributions. Repeated requests from one user, autocorrelation, shared events across windows, or rounded categorical values can invalidate a naive calibration. The empirical CDF gap remains descriptive, but the advertised p-value needs a method appropriate to the data's dependence and discreteness. A named test is not permission to ignore how the observations were generated.

Multivariate tests can compare joint structure through a chosen kernel, representation, or classifier. Freeze these choices or account for their fitting in the testing procedure; choosing a representation because it separates the very samples being tested can bias the resulting evidence. A permutation test needs exchangeability under its null, including any grouping or fitted preprocessing. Even a valid rejection establishes a distributional discrepancy under the test's assumptions, not the size of a model's accuracy loss. Report the statistic, sample sizes, reference identity, and relevant examples beside the decision.

For a fixed family of M tests, Bonferroni rejects an individual null when its valid p-value is at most α/M. The union bound controls the probability of at least one false rejection by α without requiring independence between tests. It does require each p-value to satisfy its null-validity condition. Increasing feature coverage, adding slices, or looking at additional planned windows expands the family. An engineer should therefore distinguish the number of charts displayed from the number of inferential opportunities that produced the alert.

If twenty independent null tests each reject with probability 0.05, the chance of at least one rejection is one minus 0.95 to the twentieth power. Real monitoring tests are often dependent, so that arithmetic is an illustrative independent case rather than a universal correction. Reusing a reference window does not by itself invalidate Bonferroni, but can affect the validity or dependence of individual test constructions. Practical thresholding can also require a minimum effect size, provided the combined rule and its interpretation are documented. Statistical significance alone can flag negligible movement at large volume.

An hourly monitor that resets its nominal error budget every hour does not retain the same lifetime false-alert guarantee. A finite planned horizon can include every feature–window test in one family. Alternatively, a predeclared spending sequence with a summable total error budget can support a union-bound argument when each scheduled test remains valid. Specialized sequential tests and confidence sequences offer other guarantees under their own assumptions. Stopping when a conventional fixed-sample p-value first looks small requires a justification beyond the original fixed-sample threshold.

The sampling instrument shows how null variability changes with sample size, without implementing a two-sample test. Reset, set p=0.50 and n=25, predict sample-mean variance 0.01, then Play or Step. Separately Reset, set p=0.50 and n=100, predict variance 0.0025, then Play or Step. Exact binomial masses remain fixed while playback highlights outcomes. Neither the normal overlay nor the displayed standardized-CDF bound controls a family's alert rate. Use the instrument to understand variation, then evaluate the actual detector and monitoring schedule with their stated null assumptions.