Law of Large Numbers and CLT - Core Definitions

Limit theorems describe the behavior of sequences of random variables, providing the theoretical foundation for statistical inference.

Convergence Concepts

Definition

Let $X_1, X_2, \ldots$ be a sequence of random variables and $X$ a random variable.

Convergence in Probability: $X_n \xrightarrow{P} X$ if for all $\epsilon > 0$ : $\lim_{n \to \infty} P(|X_n - X| > \epsilon) = 0$

Convergence in Distribution: $X_n \xrightarrow{d} X$ if: $\lim_{n \to \infty} F_{X_n}(x) = F_X(x)$ at all continuity points of $F_X$ .

Convergence Almost Surely: $X_n \xrightarrow{a.s.} X$ if: $P\left(\lim_{n \to \infty} X_n = X\right) = 1$

Relationships: Almost sure convergence $\Rightarrow$ convergence in probability $\Rightarrow$ convergence in distribution.

Example

Let $X_n = X + 1/n$ where $X$ is any random variable.

Then $X_n \xrightarrow{a.s.} X$ (and thus in probability and distribution) since: $|X_n - X| = 1/n \to 0 \text{ with probability 1}$

Law of Large Numbers

Theorem

Let $X_1, X_2, \ldots$ be IID with $E[X_i] = \mu$ and $\text{Var}(X_i) = \sigma^2 < \infty$ . Then: $\bar{X}_n = \frac{1}{n}\sum_{i=1}^n X_i \xrightarrow{P} \mu$

Proof Sketch: By Chebyshev's inequality: $P(|\bar{X}_n - \mu| > \epsilon) \leq \frac{\text{Var}(\bar{X}_n)}{\epsilon^2} = \frac{\sigma^2/n}{\epsilon^2} \to 0$ □

Theorem

Under the same conditions as WLLN: $\bar{X}_n \xrightarrow{a.s.} \mu$

The strong law states that sample averages converge to the population mean with probability 1, justifying the use of sample means as estimators.

Example

Monte Carlo Integration: Estimate $\int_0^1 g(x) \, dx$ by generating uniform $(0,1)$ samples $U_1, \ldots, U_n$ : $\hat{I}_n = \frac{1}{n}\sum_{i=1}^n g(U_i) \xrightarrow{a.s.} E[g(U)] = \int_0^1 g(x) \, dx$

For $g(x) = \pi x \sqrt{1-x^2}$ (quarter circle), this estimates $\pi/8$ .

Remark

The law of large numbers provides the theoretical justification for the frequentist interpretation of probability: long-run relative frequencies converge to probabilities. It's the foundation for empirical science—averages of repeated measurements converge to true values.