Observed clicks depend on exposure, position, and the policy that selected an item

One simple click model factors the chance of a click into examination at a position and a relevance-related response given examination. This can illustrate why identical items receive different click rates at different positions. Its assumptions are restrictive: examination can depend on context, neighboring items, presentation, or earlier clicks, and relevance need not be independent of how a policy assigns positions. A position-only factor fitted to observational data is not automatically the true examination probability. Randomized swaps or another justified design can provide identification evidence.

Separate the probability of choosing an action from the probability that a chosen item is actually examined. An action may be one item, an item-position pair, or an entire slate; the logging unit determines which propensity is required. Marginal item frequency is generally not the conditional action probability for a particular request context and eligible set. Persist the actual policy, candidate availability, randomization, and action definition so later evaluation does not replace the original mechanism with a convenient frequency estimate.

For logged contextual-bandit decisions, inverse propensity scoring reweights the observed reward by target-action probability divided by logging-action probability. Its usual unbiased policy-value interpretation needs correct propensities, support wherever the target acts, and an appropriate consistency and no-unmeasured-selection model. A randomized logger with recorded probabilities can establish important parts of that design. An observational policy with hidden decision inputs may not. For slates with interactions, an item-level weight can be insufficient to identify an entire slate policy's value.

Very small nonzero propensities create large weights and potentially high variance. A finite-sample estimate can lie outside the reward's usual range even when the ideal estimator is unbiased. Weight clipping and self-normalization can improve stability but alter the estimator and its guarantees; report the method and diagnostics. Zero propensity is different from a noisy small denominator: outcomes for an unsupported target action are not identified from that log without additional assumptions or new data. No finite weight can reconstruct a never-observed intervention automatically.

Audit the eligible actions, propensity distribution, maximum weights, effective coverage, reward delay, and cohort support before trusting an offline estimate. Compare alternative estimators and sensitivity analyses under documented assumptions. A randomized online experiment can provide causal evidence for its defined population and outcome, but interference, attrition, delayed reward, and changing traffic still matter. Click lift answers a click question; satisfaction, retention, and other user outcomes need their own measures and horizons. Keep uncertainty and the policy versions with any comparison that could change serving behavior.

The sampling lab illustrates a much simpler identified Bernoulli law. Reset, set both p=0.4 and n=25, predict sample-mean variance 0.0096, then Play or Step. Reset and set both p=0.4 and n=100 for variance 0.0024. It displays exact independent-binomial masses, not logged-policy reweighting or causal inference, and playback only highlights outcomes. Its normal overlay is approximate; the Berry–Esseen value concerns standardized CDFs, not each bar or an IPS estimate. Real propensity-weighted uncertainty needs the actual weighting and dependence design rather than this unweighted toy formula.