Law of large numbers
The law of large numbers is a foundational result connecting population parameters to sample statistics. It formalizes the idea that averages computed from repeated sampling stabilize around the underlying expected value. In sampling-distribution language, it describes the concentration of the sampling distribution of the sample mean \(\bar{X}\) (and, in Bernoulli settings, the sample proportion \(\hat{p}\)) around the corresponding population parameter as the sample size grows.
With independent, identically distributed observations having a finite mean \(\mu\), the sample mean \(\bar{X}_n\) approaches \(\mu\) as \(n\) increases. Random fluctuation remains present, but large sustained departures from \(\mu\) become increasingly unlikely.
Mathematical statement
Let \(X_1, X_2, \dots\) be independent and identically distributed (i.i.d.) random variables with finite expectation \(E[X_i]=\mu\). Define the sample mean \[ \bar{X}_n=\frac{1}{n}\sum_{i=1}^{n} X_i \] The law of large numbers asserts that \(\bar{X}_n\) converges to \(\mu\) as \(n\to\infty\) in a precise probabilistic sense.
Weak and strong forms
| Form | Convergence statement | Meaning in probability language |
|---|---|---|
| Weak law of large numbers (WLLN) | \(\bar{X}_n \to \mu\) in probability | For every \(\varepsilon>0\), \(P\!\left(\lvert \bar{X}_n-\mu\rvert>\varepsilon\right)\to 0\). Large deviations become rare as \(n\) increases. |
| Strong law of large numbers (SLLN) | \(\bar{X}_n \to \mu\) almost surely | With probability 1, a single infinite sequence of averages eventually stays arbitrarily close to \(\mu\). Randomness remains, but persistent divergence has probability 0. |
Both forms express stabilization of averages. The strong form is a stronger mode of convergence than the weak form. Standard sufficient conditions include i.i.d. sampling with finite \(E[\lvert X\rvert]\) for the strong law and finite variance (or other moment conditions) for many common weak-law proofs.
Bernoulli trials and sample proportion
A central special case uses Bernoulli random variables. Let \(X_i\in\{0,1\}\) with \(P(X_i=1)=p\). The sample proportion of successes is \[ \hat{p}_n=\bar{X}_n=\frac{1}{n}\sum_{i=1}^{n}X_i \] The law of large numbers implies \(\hat{p}_n\to p\), meaning relative frequencies stabilize around the underlying probability \(p\) as the number of trials grows.
Visualization of convergence of a running mean
Finite-sample accuracy statements
The law of large numbers is an asymptotic guarantee; finite-sample behavior is described by probability bounds. Under i.i.d. sampling with finite variance \(\sigma^2=\operatorname{Var}(X)\), Chebyshev’s inequality implies \[ P\!\left(\lvert \bar{X}_n-\mu\rvert \ge \varepsilon\right)\le \frac{\sigma^2}{n\varepsilon^2} \] This bound exhibits the characteristic \(1/n\) shrinkage in tail probability for fixed \(\varepsilon\).
For bounded variables such as Bernoulli trials, sharper exponential bounds are available. For \(X_i\in[0,1]\) with \(E[X_i]=p\), Hoeffding’s inequality gives \[ P\!\left(\lvert \hat{p}_n-p\rvert \ge \varepsilon\right)\le 2e^{-2n\varepsilon^2} \] which quantifies how quickly relative frequency concentrates around \(p\).
Common misconceptions and pitfalls
Law of large numbers versus central limit theorem
The law of large numbers concerns convergence of the sample mean to \(\mu\). The central limit theorem concerns the approximate normal shape of the standardized sampling distribution of \(\bar{X}_n\) for large \(n\). Stabilization of \(\bar{X}_n\) and normal approximation address different questions.
Randomness does not disappear
Large samples reduce relative fluctuation of averages, but individual outcomes remain random. Long runs or short-term imbalance remain possible even when the long-run proportion converges.
Condition failures
Independence and identical distribution are structural assumptions in common formulations. Heavy-tailed models with infinite mean, strong dependence, or changing distributions can violate the usual conclusions for \(\bar{X}_n\).