Collect enough support for the failures an aggregate can hide

Let a slice be a predeclared condition on requests, such as language plus tool type. Its prevalence is the probability a target-population request belongs to it. Its conditional failure rate is another quantity, measured among eligible slice requests. Multiplying the two gives the population contribution of that failure only when definitions and conditioning match. Overlapping slices require care: summing their contributions can double-count requests. Preserve intersection counts and grouping identifiers rather than report many apparently precise slice percentages drawn from the same few conversations.

Under independent identical population draws with slice probability r, the chance of collecting no slice examples in n requests is (1−r)^n. An expected count of one does not guarantee one observed example, much less reliable conditional evaluation. Stratified oversampling can increase slice support, but retain its selection design and target weights when estimating population aggregates. Adversarially authored cases are useful for capability or failure discovery; their construction is usually not a random draw from the natural slice, so their pass rate should not silently become a deployment frequency estimate.

If a fixed-size independent slice sample has zero failures, the observed fraction is zero but the population failure probability remains uncertain. Inverting the probability of zero failures gives a one-sided exact binomial upper confidence bound. The bound is larger when support is smaller. It applies to the declared fixed sample under the stated Bernoulli model, not automatically to duplicated variants, a hand-selected easiest subset, or a sequence stopped the first time the model appears safe. Record the denominator and selection process with every zero-failure claim.

The confidence level is a property of the interval procedure under repeated valid sampling, not a probability that a fixed unknown rate lies below the particular realized endpoint without an additional Bayesian model. A zero-failure bound for one slice does not provide simultaneous coverage for many slices at the same nominal level. Allocate joint uncertainty deliberately when the release depends on several statements. More samples can narrow a bound for the same population; they do not establish that the slice definition captures all relevant risks or that next month's traffic has the same distribution.

For each important slice, state the required support, allowed failure consequences, uncertainty method, and action when evidence is missing. A missing slice should be reported as unassessed rather than folded into a passing overall score. Use targeted data collection, shadow evaluation, or a narrower deployment scope when that matches the decision. Discovery of a new failure can justify adding regression cases, while a separate untouched sample is needed for a fresh confirmatory rate claim after model or prompt tuning. Report both roles rather than mixing their denominators.

The probability lab models rare membership only. Reset, set both p=0.02 and n=30, predict its expected count and zero-success bar, then inspect. The expected count is 0.6 and the no-member probability is 0.98^30≈0.54548. The thirty-trial limit is the actual control range; the instrument does not collect a thousand cases or estimate a conditional failure rate. Use the display to see why a small random test often lacks rare-slice support, then design the real collection and inference process explicitly.