Monitor feature populations before interpreting a drift score

Store the reference dataset identifier, capture interval, eligibility rules, sampling probabilities, schema, and transformation version with its statistics. Comparing a weekday production window with an unbalanced weekend training sample can mostly measure a known calendar effect. Maintain a stable training reference for deployment mismatch and a recent, seasonally appropriate production reference for operational changes. These answer different questions. A rolling reference alone can absorb a slow degradation until each adjacent comparison appears small, so preserve the historical context needed to recognize accumulated movement.

Count the requests that should have been captured as well as the records that arrived. Sampling after a successful response excludes failures; sampling one request per session differs from sampling every request. If logging probabilities differ across groups, either define the target as the captured population or use justified inclusion weights and report their support. Data minimization can restrict what is retained, but a feature summary without its population contract cannot distinguish changed behavior from changed observation. Track duplicate identifiers and capture gaps beside distribution statistics.

For categorical features, align both windows to the union of named categories, including new categories with zero reference count. Total variation summarizes the amount of probability mass that would have to move to match the histograms. Keep the largest category contributions so the aggregate distance remains explainable. Numeric bins need fixed, versioned boundaries and declared underflow, overflow, missing, and nonfinite policies. Recomputing quantile boundaries independently in each window can hide a physical scale change by giving every new bin approximately equal occupancy.

Missingness deserves its own denominator and breakdown by upstream source. A mean computed only over present values can remain constant while half the requests lose a feature. Monitor valid count, missing fraction, ranges, and selected quantiles together rather than treating one statistic as a complete detector. Distances calculated from finite samples also fluctuate under an unchanged population. An operational threshold should reflect sample size, expected seasonality, investigation cost, and model sensitivity; the value of a distance is not automatically a calibrated probability of harm.

Two binary features can each remain balanced while their relationship reverses. If the reference contains only equal pairs and the current population contains only unequal pairs, both individual histograms are unchanged but their joint support is different. Add selected cross-feature checks where the model or domain suggests meaningful interactions. Multivariate representations and two-sample procedures can help, but their sensitivity depends on the representation and available examples. A learned embedding can discard exactly the distinction that a downstream decision needs, so passing an embedding monitor is limited evidence.

The sampling instrument isolates ordinary proportion uncertainty. Reset, set p=0.20 and n=25, predict mean 0.20 and sample-mean variance 0.0064, then Play or Step. Separately Reset, set p=0.20 and n=100, predict the same mean and variance 0.0016, then Play or Step. Playback highlights exact binomial outcomes rather than collecting data. This comparison motivates recording sample size, while providing no multivariate detector, seasonal adjustment, or missing-data correction. Production interpretation still requires tracing the changed categories to their originating requests.