Consistency and uniform laws of large numbers

Write $P_n f=n^{-1}\sum_i f(X_i)$ and let $Q_n(\theta)=P_n\rho_\theta$ approximate $Q(\theta)=E\rho_\theta(X)$. A pointwise law fixes a deterministic parameter before drawing data. By contrast, an empirical minimizer can seek whichever location has the most favorable random error. Uniform convergence bounds that error simultaneously over the whole parameter set, so it remains valid when evaluated at the selected estimator. This is the precise reason a uniform law enters estimation.

Suppose the reported estimator is an approximate minimizer: its empirical objective exceeds the infimum by a nonnegative error tending to zero in probability. Comparing it with the population minimizer bounds its population excess risk by twice the uniform error plus this optimization error. This argument does not require differentiability, a unique empirical optimizer, or an exact numerical solution. It does require the population minimizer to belong to the comparison set and all criteria used in the comparison to be finite.

For each positive distance, assume the population criterion outside that neighborhood exceeds its minimum by a strictly positive amount. Whenever the preceding excess-risk bound falls below this gap, the estimator must lie inside the neighborhood. Thus uniform convergence plus separation proves convergence in probability of the parameter. Compactness, continuity of the population objective, and uniqueness of its minimum are one convenient route to separation. They are sufficient conditions, rather than part of the definition of consistency.

A common uniform-law theorem assumes a compact finite-dimensional parameter set, almost-sure continuity of the loss in the parameter, and an integrable envelope dominating the absolute losses. A finite-cover argument and dominated convergence then reduce the uniform problem to finitely many ordinary laws of large numbers. Bounded parameters alone do not bound an unbounded observation: squared loss still needs an appropriate moment, such as a finite second moment. Noncompact problems need additional localization or coercivity arguments instead of an unjustified appeal to compactness.

If the population criterion has several minimizers, a perfect empirical approximation cannot decide which one is scientifically intended. One may obtain convergence to the minimizing set, but convergence to a named member needs a further identifying restriction or a selection rule with its own analysis. Even an algorithm returning a global optimum cannot repair this ambiguity. The practice problem makes the distinction exact by eliminating all sampling error while retaining two different parameter values with identical loss.

Consistency also says nothing by itself about a convergence rate or the accuracy of a normal confidence interval. A criterion can be uniformly close yet extremely flat near its minimum. Later lessons add local curvature, stochastic equicontinuity, and central limit theorems to obtain rates and distributions. At this stage, check the three logically separate ingredients: empirical approximation, objective optimization, and population identification. A failure in any one of them invalidates the simple argmin proof.