Unbiased estimation and Rao–Blackwell improvement

An estimator $U$ of $g(\theta)$ is unbiased when its expectation equals the target for every parameter value. Under squared-error loss, mean squared error decomposes into variance plus squared bias whenever the second moment exists. Unbiasedness therefore makes risk equal variance, but does not guarantee the smallest risk among all estimators. A mildly biased estimator may reduce variance enough to improve mean squared error for some or all parameter values. Always name both the target and the loss before describing an estimator as better.

For iid Bernoulli observations, the first observation and the sample mean are both unbiased for the common probability. The first observation has variance $p(1-p)$, while the sample mean has variance $p(1-p)/n$. Independence makes the cross-covariance terms vanish in the latter calculation. Repeated copies of one observation would not gain this reduction. Finite-sample variance formulas here are exact under the iid model; no normal approximation or large-sample argument is being used.

Let $T$ be sufficient and let $U$ have finite second moment under each model law. Define $V=E[U\mid T]$ using a parameter-free conditional law supplied by sufficiency. This common version is an actual statistic, computable without knowing the parameter. Iterated expectation preserves the mean, so an unbiased $U$ produces an unbiased $V$. Without sufficiency, a conditional expectation may depend on the unknown parameter and fail to define an available estimator even though the formal variance identity remains true for each fixed law.

Total variance splits the original variance into the variance of the conditional mean plus the average conditional variance. The second term is nonnegative, proving the improvement. Equality at a parameter value occurs precisely when the original estimator already equals its conditional expectation almost surely under that law. More generally conditional Jensen gives no increase in risk for convex loss as a function of the reported action, when the relevant expectations and conditional mean exist. This is a finite-sample theorem, not an asymptotic efficiency slogan.

If the sufficient statistic is also complete, any two unbiased integrable functions of it estimating the same target agree almost surely under every model law. Subtract them: the difference is a function of the statistic with zero expectation everywhere, so completeness forces that difference to vanish. Combining this uniqueness with Rao–Blackwell gives the Lehmann–Scheffé conclusion: an unbiased function of a complete sufficient statistic with finite variance is uniformly minimum variance among all unbiased finite-variance estimators of that target.

Uniformly means the variance comparison holds at each parameter value, not merely on average over a prior. Minimum variance is restricted to unbiased estimators; it does not rule out a biased estimator with a better chosen risk tradeoff. Existence also matters: some targets have no unbiased estimator, or no one with finite variance. In practice, identify a useful starting estimator, find a sufficient statistic, compute the conditional mean, and then check completeness if you want a uniqueness and optimality claim. Each step supplies a distinct part of the argument.