Keep eligibility, assignment, and actual intervention separate

Eligibility describes whether a unit satisfies the enrollment rule. Assignment records its randomized arm. Exposure or intervention records the point where the changed decision could affect the unit, and outcomes record subsequent behavior under a declared window. A model being selected in a configuration lookup is not necessarily the model response displayed to the user. Server errors, fallback paths, caches, and client rendering can separate selection from delivery, so choose the boundary that matches the hypothesis.

An exposure ledger should carry the experiment and configuration IDs, randomization-unit ID, assigned arm, actual model artifact or fallback identity, event ID, event timestamp, and relevant request/session linkage. Deduplicate retries by logical event identity and preserve both event and ingestion time. A many-to-many join from outcomes to repeated exposure rows can inflate numerators. Reconcile assignment and exposure funnels by arm before interpreting lift, including dropped or malformed events.

If treatment changes whether a unit reaches or completes the exposure event, exposed treatment and exposed control units need not be comparable. A faster model might retain additional marginal users who would have left under control. Comparing only the users who remained changes the composition in a treatment-dependent way. The assignment-based intention-to-treat comparison includes that reach effect, whereas the exposed-only difference can have another sign without any contradiction.

A triggered analysis can improve sensitivity when the trigger is a valid pre-intervention condition defined comparably in both arms. Counterfactual logging in control can record whether a request would have qualified for a treatment feature, as Microsoft’s experimentation guidance discusses. This requires correct symmetric logic and a defensible trigger; merely inventing a control exposure row does not remove selection bias. Keep the assignment-based result and check trigger coverage rather than discarding unaffected users opportunistically.

A missing exposure record can mean no intervention, instrumentation loss, or an unjoined identifier. Preserve a reason or diagnostic pathway rather than automatically treating every missing record as an unexposed unit. Compare source counts, join coverage, event lag, and fallback identity by arm. If the changed model crashes before emitting its own log, using only successful response logs systematically removes precisely the failures the experiment should measure.

Freeze the event schema and attribution rules with the experiment definition. Changes to logging logic during a run create analysis epochs that need investigation rather than automatic concatenation. A mature pipeline can reproduce a unit’s assignment, qualifying trigger, actual serving version, and attributed outcome from durable IDs. This makes a surprising result debuggable: the team can ask whether behavior changed, delivery changed, or the observation process changed instead of treating every dashboard difference as model quality.