Randomize the entity whose outcome you intend to compare

Suppose the goal is conversion per person over seven days. Randomizing each request independently can mix experiences within one person and produce carryover from earlier requests. Randomizing a stable user identifier better matches that intervention, provided the identifier actually links the relevant activity. Device IDs answer another question when one person uses several devices or several people share one device. Neither identifier is universally correct; the target experience and identity limitations determine the choice.

The analysis must respect dependence induced by assignment. A user with many requests does not become many independent randomized users. Aggregate the declared user outcome or use an estimator and uncertainty calculation that accounts for repeated observations within users. A request-weighted metric can be legitimate when it is the intended estimand, but copying a user-level conversion label onto every request changes the weights and can change the apparent effect without adding new information.

A common implementation hashes the experiment identifier, a stable unit identifier, and a versioned salt into a bucket. A fixed allocation rule converts that bucket into an arm. Repeated checks then return the same result as long as the identifiers, salt, and rule remain unchanged. The hash implements reproducibility; approximate uniform independent bucketing across units is a design assumption that should be checked, rather than a property established by printing a few balanced examples.

Changing the threshold, targeting rule, identifier resolution, or salt can change assignments even if the hash function is unchanged. Persistent assignment can retain the first enrolled decision while enrollment rules evolve, but its behavior must be specified and logged. A ramp should record configuration epochs and distinguish newly eligible units from previously assigned units. Reclassifying an existing unit after observing its outcome damages the original randomized comparison and cannot be fixed by relabeling the chart.

Store the randomization-unit type, identifier version, assignment probability, configuration epoch, and first-assignment timestamp with the experiment record. Test retries and sign-in transitions explicitly. If anonymous and signed-in IDs are joined, define whether an existing assignment wins and how conflicting histories are handled. An identity merge performed after treatment can otherwise turn one person into two counted units or selectively discard an arm’s earlier behavior.

An A/A exercise can expose integration and measurement problems before a substantive model comparison, but a passing A/A run does not prove future experiments correct. Keep assignment diagnostics separate from outcome significance. When treatment changes activity volume, a growing request log is especially easy to misread as a growing independent sample. Preserve the chosen denominator and unit-level lineage so the reported uncertainty still describes the entity that was actually randomized.