Track slice quality with support and composition visible

Choose slices from the intended deployment question: language, device family, request source, input length, or an operationally meaningful difficulty category. A disjoint exhaustive partition allows aggregate error to be written as the sum of slice errors weighted by traffic shares. Report both quantities over time. Comparing the new traffic shares with old within-slice rates gives one composition-only counterfactual; comparing within-slice rates on a fixed mixture highlights another aspect. Neither standardization automatically identifies a causal effect of the model when users, labels, or policies also change.

Many useful slices overlap. A request can belong to both a language slice and a device slice, so adding their counts or weighted errors would double-count it. Keep each slice's denominator explicit and use a genuine partition only when applying the aggregate identity. Version the slice predicates, including treatment of unknown categories and missing metadata. If the group assignment itself changes after a preprocessing release, an apparent quality jump may simply reflect requests moving between groups. Preserve enough counts to reconcile the overall report with its partitioned view.

Report resolved examples, unique users or clusters, outcome completeness, and the metric's numerator and denominator. Precision needs predicted positives; recall needs actual positives. A large total slice count does not imply either denominator is large. Under independent identical Bernoulli errors, the standard error of an empirical error proportion is approximately the square root of p(1−p)/n, with an estimated p often substituted. Small or extreme samples need a more suitable interval construction than a naive symmetric normal interval. Repeated users and temporal correlation require dependence-aware analysis.

When examples carry nonnegative weights, a common diagnostic is squared total weight divided by squared-weight sum. It reports weight concentration: a few large weights can leave much less information than the raw row count suggests. This effective-size expression does not, on its own, correct clustering, estimated propensity uncertainty, label bias, or arbitrary metric variance. Report the weight policy and largest contributions as well as the diagnostic. If a slice has no resolved outcomes or no relevant positives, show the metric as unavailable rather than manufacturing a zero or a perfect score.

Predeclare critical slices and minimum evidence requirements for release decisions. Exploratory searches over many combinations can discover problems, but the most extreme observed group also benefits from selection by chance; confirm it on new evidence or use an appropriate simultaneous procedure. Minimum-count suppression should preserve an explicit 'insufficient evidence' state, not silently remove the group from the dashboard. A rare group can deserve targeted data collection or a conservative fallback before it has enough observations to support a precise performance claim.

The sampling lab isolates the effect of support. Reset, set p=0.10 and n=20, predict sample-mean variance 0.0045, then Play or Step. Separately Reset, set p=0.10 and n=100, predict variance 0.0009, then Play or Step. Playback highlights exact independent Bernoulli outcomes, rather than resampling a slice. The normal overlay can be poor near sparse boundaries; its CDF error bound does not certify individual bars or an arbitrary subgroup interval. Actual slice monitoring must also account for weighting, dependence, label availability, and how the slice was selected.