Asymptotic normality of M-estimators

Assume iid observations. Let the population loss have an interior minimizer, and suppose the estimator converges to it. In a fixed neighborhood, assume the loss is twice differentiable in the parameter, the gradient at the target has mean zero and finite covariance, and empirical Hessians converge uniformly in probability to a continuous population Hessian. Its value at the target must be nonsingular. Suitable domination justifies the population differentiations. These assumptions are stronger than necessary, but they make each step of the expansion transparent.

The optimization residual must also be small enough: the empirical gradient at the reported estimator is negligible compared with the inverse square root of sample size. An objective error merely tending to zero was enough for consistency, but does not imply this gradient condition or the desired asymptotic distribution. Integrating the Hessian along the segment from the target to the estimator gives a valid vector expansion. There is no need to claim that one intermediate point supplies a simultaneous multivariate mean-value identity.

The multivariate central limit theorem applies to the centered gradients. Their covariance is a measure of observation noise, whereas the expected Hessian describes how strongly the criterion changes when the parameter moves. These are different mathematical quantities. Under the stated smoothness and consistency conditions, the limiting covariance sandwiches the gradient covariance between inverse curvature matrices. The transpose on the right matters when the derivative matrix is not symmetric, and the same formula later applies to estimating equations.

For a correctly specified regular likelihood, the information identity can equate gradient covariance and expected negative-log-likelihood curvature. That additional identity simplifies the sandwich to inverse Fisher information. A working likelihood fitted to another distribution need not satisfy it. The population optimum is then a pseudo-true parameter for that loss, and inference must target that parameter explicitly. Replacing the sandwich by inverse curvature under misspecification can either overstate or understate uncertainty, even when the fitted parameter itself remains consistent.

A singular Hessian removes the local inverse used in this proof. A boundary optimum can satisfy an inequality instead of a zero-gradient equation, producing a projected or truncated limit. Heavy tails may invalidate the score central limit theorem even when a population objective exists. Nonsmooth losses, including absolute deviation, often remain asymptotically normal, but their justification uses a local stochastic expansion rather than an ordinary sample Hessian. These are different reasons to check the theorem instead of relying on the estimator's familiar name.

The expansion also suggests how to estimate uncertainty: evaluate empirical versions of curvature and gradient covariance near the fitted parameter, then assemble the sandwich. Their consistency needs its own law of large numbers and moment control. A numerical Hessian alone is not a complete uncertainty estimate. In a report, specify the target, the loss, the independence assumption, and whether correct likelihood specification was invoked. That information makes the claimed standard errors reproducible and exposes where a robust covariance estimate is necessary.