Estimate a policy from logged actions only where identification permits

Let a context X describe information available before choosing an action A, and let R(a) be the reward that would follow action a. Consistency links the observed reward to the chosen action’s potential reward. A randomized logging policy can make action choice independent of potential rewards conditional on the logged information. Observational logs need a conditional no-unmeasured-confounding assumption instead; unlogged factors that influence both action choice and reward can invalidate the same formulas.

Positivity requires the logger to choose every action the target policy may choose in each relevant context. A deterministic logger lacks that support for alternative actions unless further modeling assumptions are introduced, and a reward model’s extrapolation is not nonparametric identification. Also specify the context distribution, reward horizon, missingness, and interference conditions. A one-step contextual-bandit estimator does not automatically evaluate a policy whose actions change future contexts and long-term trajectories.

Doubly robust evaluation starts with the reward model’s target-policy prediction and corrects it using the observed action’s weighted residual. When the propensity model is correct, the residual correction removes reward-model bias in expectation under the identification conditions. When the reward model is correct, residuals have conditional mean zero, supporting robustness to some propensity misspecification. With estimated nuisance models, consistency and uncertainty need the relevant fitting, convergence, and sample-splitting conditions; the slogan alone is not a finite-sample guarantee.

Known logged randomization probabilities are especially valuable because they document the action mechanism rather than reconstructing it later from an outcome-contaminated dataset. Preserve probabilities at decision time, including eligibility and action-set restrictions. If the logger adapts over time, its conditional probability may depend on prior history as well as current context. Use that actual conditional probability and an appropriate adaptive-analysis method rather than replacing it with an overall action frequency.

Small logging probabilities create large weights and high variance. Inspect support by context and action, tail weights, reward completeness, and the contribution of individual records. Weight clipping or self-normalization can improve stability but generally changes finite-sample bias properties; report the exact estimator and sensitivity to those choices. A precise-looking number can be dominated by very few rare exploratory actions, so sample size alone is not a reliable information measure.

Keep policy selection separate from final evaluation. Choosing the best policy among many on the same noisy log creates selection optimism, even if each fixed policy estimator is valid. Use held-out evaluation or a method that accounts for the search. Finally, an offline estimate concerns the specified logged-context population and reward definition; a live experiment is still needed when deployment changes exposure, interference, latency, or the distribution of future user behavior.