Common Distributions - Main Theorem

The Central Limit Theorem is perhaps the most important result in probability theory, explaining why the normal distribution appears ubiquitously in nature and justifying many statistical procedures.

Central Limit Theorem (CLT)

Theorem

Let $X_1, X_2, \ldots, X_n$ be independent and identically distributed random variables with mean $\mu$ and finite variance $\sigma^2$ . Define: $S_n = \sum_{i=1}^n X_i, \quad \bar{X}_n = \frac{S_n}{n}$

Then as $n \to \infty$ , the standardized sum converges in distribution to the standard normal: $Z_n = \frac{S_n - n\mu}{\sigma\sqrt{n}} = \frac{\bar{X}_n - \mu}{\sigma/\sqrt{n}} \xrightarrow{d} \mathcal{N}(0,1)$

Equivalently, for any $x$ : $P(Z_n \leq x) \to \Phi(x) = \frac{1}{\sqrt{2\pi}} \int_{-\infty}^x e^{-t^2/2} dt$

This remarkable result states that sums (or averages) of IID random variables become approximately normal, regardless of the original distribution!

Example

Roll a fair die $n = 100$ times. Each roll has mean $\mu = 3.5$ and variance $\sigma^2 = 35/12 \approx 2.917$ .

Total sum $S_{100}$ has: $E[S_{100}] = 100(3.5) = 350, \quad \text{Var}(S_{100}) = 100(35/12) \approx 291.7$

By CLT, $S_{100} \approx \mathcal{N}(350, 291.7)$ .

Probability total is between 330 and 370: $P(330 < S_{100} < 370) \approx \Phi\left(\frac{370-350}{\sqrt{291.7}}\right) - \Phi\left(\frac{330-350}{\sqrt{291.7}}\right)$ $\approx \Phi(1.17) - \Phi(-1.17) \approx 2\Phi(1.17) - 1 \approx 0.758$

Conditions and Extensions

Lindeberg-Lévy CLT: The basic form requires IID with finite variance.

Lyapunov CLT: Allows non-identical distributions under certain growth conditions on moments.

Berry-Esseen Theorem: Quantifies the rate of convergence: $\sup_x |P(Z_n \leq x) - \Phi(x)| \leq \frac{C\rho}{\sigma^3\sqrt{n}}$

where $\rho = E[|X - \mu|^3]$ and $C \approx 0.4748$ . This bounds the approximation error.

Example

For Bernoulli $(p)$ : $\mu = p$ , $\sigma^2 = p(1-p)$ , $\rho = p(1-p)(1-2p)^2 + p^2(1-p) + p(1-p)^2$ .

For $p = 0.5$ and $n = 30$ : $\text{Error} \leq \frac{0.4748 \times 0.125}{0.125\sqrt{30}} \approx 0.089$

The normal approximation is accurate to within about 9%.

Applications

Statistical Inference: The CLT justifies using normal-based confidence intervals and hypothesis tests for sample means, even when the population distribution is non-normal.

Quality Control: If individual measurements have mean $\mu$ and variance $\sigma^2$ , the average of $n$ measurements has approximately: $\bar{X} \sim \mathcal{N}\left(\mu, \frac{\sigma^2}{n}\right)$

The standard error $\sigma/\sqrt{n}$ decreases with sample size.

Example

A factory produces bolts with mean length 10cm and std dev 0.5cm (unknown distribution). Taking sample of $n = 25$ bolts: $\bar{X} \approx \mathcal{N}\left(10, \frac{(0.5)^2}{25}\right) = \mathcal{N}(10, 0.01)$

Standard error: $0.5/\sqrt{25} = 0.1$ cm.

95% confidence interval for population mean: $\bar{x} \pm 1.96(0.1) = \bar{x} \pm 0.196$

If sample mean is 10.15cm, we're 95% confident true mean is in $[9.954, 10.346]$ .

Remark

The CLT explains why the normal distribution dominates statistics: many measurable quantities are sums or averages of independent effects. Heights, test scores, measurement errors—all tend toward normality due to the CLT. This theorem is the theoretical foundation for the ubiquity of the Gaussian distribution in nature and statistics.