Use allocation imbalance to investigate the observation process

Start from distinct randomization units and the correct allocation probabilities for the cohort and configuration epoch. A planned 70/30 experiment should not be tested against 50/50. If allocation changes over time, expected counts must account for those changes and enrollment rules rather than applying the final setting to the entire history. Fixed-size constrained assignment has a different count distribution from independent Bernoulli assignment, so the test must match the mechanism.

For a simple independent multinomial allocation with fixed probabilities and sufficiently large expected cell counts, Pearson’s chi-square statistic compares observed and expected counts. With two arms and no fitted probability parameters, its large-sample reference distribution has one degree of freedom. An exact binomial procedure can be appropriate for small two-arm counts. Configure diagnostic thresholds and repeated-check behavior deliberately; a familiar p-value cutoff is not universal across platforms or monitoring frequencies.

Compare enrollment, assignment, intervention logs, joined outcome tables, and analysis filters in sequence. If the raw assignment ledger is balanced but the exposed subset is not, the cause may be exposure behavior, telemetry loss, or a treatment-dependent trigger. If a join duplicates only one arm’s high-activity users, the mismatch can arise after assignment even though the bucketing service is correct. Segment by stable pre-treatment attributes and time to locate the first divergence.

An SRM alert is evidence inconsistent with the specified counting model at its calibrated error level; it does not name the root cause or prove the direction of outcome bias. A highly engaging treatment can cause a bot filter to remove more treated users, for example, but another incident can remove low-engagement users instead. Reweighting counts to the intended ratio without understanding that selection does not generally recover the missing outcomes or the randomized estimand.

Equal counts can coexist with systematic contamination, swapped labels, wrong model versions, outcome duplication in both arms, or interference. A passing SRM check therefore cannot establish causal identification or measurement correctness. Combine it with identity, exposure, timing, and metric-quality diagnostics. Likewise, a failed check should lead to a documented resolution or a clearly limited analysis rather than a claim that a chosen outcome must be false.

Keep the diagnostic separate from the outcome stopping rule. Repeatedly searching many segments for tiny SRM p-values creates its own multiplicity and triage problem, while a single global aggregate can hide offsetting local failures. Record the configured expectation, population boundary, test, threshold, and investigative evidence. If a pipeline fix changes the analyzed population, recompute the relevant result from traceable source data and disclose the corrected cohort instead of patching only the final dashboard ratio.