Fisher information and Cramér–Rao bounds

For a differentiable scalar density at an interior parameter value, the score is the derivative of its log density with respect to the parameter. Assume a common parameter-independent support, justified differentiation under the integral, and a square-integrable score with positive finite information. Differentiating total probability one shows that the score has expectation zero. Its second moment is Fisher information. If a second differentiation is justified as well, information equals the negative expected second derivative of log density. These equalities are regularity conclusions rather than automatic symbolic rules.

For independent observations, joint log likelihood is a sum, so the joint score is a sum. The individual scores have mean zero and are independent; their cross moments vanish. Joint information therefore adds, giving $n$ times one-observation information in an iid experiment. Correlated measurements generally do not have this simple additive information. Changing parameter coordinates also changes the numerical information: applying the chain rule multiplies it by the square of the derivative of the old parameter with respect to the new one. Information must be interpreted together with its coordinate.

Let $U$ be a parameter-free statistic unbiased for a differentiable target $g(\theta)$, with finite variance. Assume differentiation of its expectation is legitimate and equals integration against the density derivative. Differentiating unbiasedness gives $g'(\theta)=E_\theta[US_\theta]$. Because the score has mean zero, this is the covariance of estimator and score. Cauchy–Schwarz bounds its square by estimator variance times score variance, yielding the scalar Cramér–Rao inequality. The proof explicitly uses both unbiasedness and differentiation conditions.

Equality in Cauchy–Schwarz requires the centered estimator to be a scalar multiple of the score almost surely at the parameter value. A lower bound need not be attainable by an available estimator, so calling every estimator above it inefficient can obscure an unattainable target. For biased estimators the derivative of their mean replaces the target derivative in the covariance calculation; the simple unbiased formula cannot be applied unchanged. With nuisance parameters the appropriate matrix information and target gradient are needed, rather than pretending other unknown coordinates are known.

Consider iid uniform observations on $[0,\theta]$. Differentiating only the interior density gives the formal joint log derivative $-n/\theta$, a nonzero constant. Its expectation is not zero because the support moves with the parameter and the missing boundary term matters. Consequently the regular score proof above does not apply. The sample maximum estimates this endpoint with uncertainty of order $1/n$, unlike the usual regular mean-estimation scale. This is a model feature, not a contradiction of a valid theorem.

Bernoulli probabilities at zero and one also lie on the parameter-space boundary, where interior formulas containing $1/[p(1-p)]$ cannot simply be evaluated. Inside the open interval the model is regular and the sample mean exactly attains the unbiased bound. Statements about approximate normal confidence intervals still need a separate finite-sample or asymptotic argument. Attaining an information bound does not prove that the estimator's distribution is normal, and an exact variance expression does not automatically justify a Gaussian tail probability.