Remove predictable variation without adjusting away treatment

Let Y be the outcome and X a preperiod covariate observed before intervention. Subtract θ times the deviation of X from a common center c. Using the same coefficient and center in both arms makes the adjusted difference equal the raw outcome difference minus θ times the baseline imbalance. The center cancels from that contrast. For a fixed coefficient and a randomized baseline unaffected by treatment, the expected baseline difference is zero, preserving the assignment effect.

This does not mean every realized adjusted estimate equals the raw estimate, or that adjustment always moves a result toward zero. If treatment happened to receive unusually active users, adjustment may reduce an inflated observed difference; the opposite imbalance may increase it. Report the adjustment specification and uncertainty rather than choosing whichever version looks more favorable. Use the randomization unit’s covariate and outcome aggregation, avoiding a user/request mismatch hidden inside a regression table.

For a single population with finite second moments, the residual variance is a quadratic function of θ. Differentiating it gives covariance divided by baseline variance as the variance-minimizing coefficient. At that coefficient, the remaining variance is the original outcome variance times one minus squared correlation. This is a population identity, not a guarantee that an estimated coefficient on a small noisy sample improves every experiment or every arm.

Estimate adjustment parameters using a documented procedure, such as independent historical data, a prespecified pooled approach with appropriate asymptotic analysis, or cross-fitting when the method requires it. Fitting a flexible predictor and evaluating it on the same records can create overfitting and invalid uncertainty. Strong baseline predictiveness helps, but the analysis must still account for coefficient estimation, heterogeneous treatment effects where relevant, and any clustering induced by assignment.

A feature computed from the experiment’s first day is not a preperiod covariate merely because it is available before the final analysis. If treatment affects it, adjusting for it can remove part of the causal pathway or create selection bias. Freeze source cutoffs and feature lineage, and verify that the baseline calculation cannot read outcome-window events through late joins, backfills, or a reused feature table whose contents changed during the run.

New users may lack history. Specify imputation and missingness indicators before inspecting treatment results, and preserve them symmetrically across arms. A complete-case analysis can change the target population, particularly if history availability is inferred from treatment-affected activity. Ratio metrics and repeated-event metrics also need a suitable unit-level or delta-method treatment; applying a simple scalar adjustment to arbitrary event ratios does not automatically inherit the elementary CUPED variance identity.