Laws of large numbers

Suppose the observations are independent and identically distributed, with mean μ and finite variance σ². Linearity of expectation makes their average unbiased. Independence removes the cross-covariances, leaving variance σ² divided by the sample size. Chebyshev’s inequality then bounds the probability of an error exceeding any fixed tolerance. The bound tends to zero, proving convergence in probability. Pairwise uncorrelated observations with a common mean μ and a common finite variance would already suffice for this particular calculation.

The bound is a guarantee rather than an exact tail probability, and can be conservative. It shows the characteristic square-root relation: reducing the root-mean-square error by a factor of two requires four times as many independent observations. The numerator contains variance, so this proof cannot be used unchanged when the variance is infinite. That limitation belongs to the proof; a weak law can still hold for independent identically distributed integrable variables with infinite variance.

The strong law for independent identically distributed real variables assumes finite expected absolute value and concludes almost sure convergence of the averages to the mean. A finite variance is sufficient but unnecessary. Almost sure convergence means that nearly every complete observation sequence eventually stays within every fixed tolerance. There is no single deterministic sample size after which all possible sequences become accurate, and no assertion that every finite prefix improves on the one before it.

Even in the finite-variance case, summing the preceding Chebyshev bounds over every sample size gives a divergent harmonic series, so the first Borel–Cantelli lemma cannot finish the proof directly. One route uses Kolmogorov’s maximal inequality at dyadic sample sizes. For centered partial sums, the probability that any of the first 2^m sums exceeds ε2^m is at most σ²/(ε²2^m), which is summable in m. Borel–Cantelli then controls whole dyadic blocks. Extending the theorem to merely integrable observations requires a truncation argument.

If every observation equals one shared Bernoulli variable Y, the average equals Y at every sample size. Its variance never decreases. Each marginal distribution is still identical, but the observations are perfectly dependent. Some dependent sequences obey other laws of large numbers, for example under suitable stationarity and ergodicity assumptions, but those are additional theorems rather than consequences of the independent-observation calculation.

An expectation must also exist in the relevant sense. A symmetric Cauchy law has no finite expected absolute value and no ordinary mean; a symmetry argument does not create one. Averages of independent standard Cauchy variables remain standard Cauchy, so the usual stabilization claim fails. In applications, inspect what constitutes an independent observation, whether measurements share a common shock, and whether extreme values are controlled. A precise theorem cannot repair a misspecified sampling model.