Probability Spaces and Events - Main Theorem

Bayes' theorem is the cornerstone of statistical inference, providing a systematic method for updating beliefs in light of new evidence. It connects forward probabilities (from causes to effects) with inverse probabilities (from effects to causes).

Bayes' Theorem

Theorem

Let $(\Omega, \mathcal{F}, P)$ be a probability space, and let $\{B_1, B_2, \ldots, B_n\}$ be a partition of $\Omega$ with $P(B_i) > 0$ for all $i$ . For any event $A$ with $P(A) > 0$ : $P(B_j|A) = \frac{P(A|B_j) P(B_j)}{\sum_{i=1}^n P(A|B_i) P(B_i)}$

Proof Outline: Starting from the definition of conditional probability: $P(B_j|A) = \frac{P(A \cap B_j)}{P(A)} = \frac{P(A|B_j) P(B_j)}{P(A)}$

Applying the law of total probability to the denominator: $P(A) = \sum_{i=1}^n P(A|B_i) P(B_i)$

Substitution yields the result. □

Interpretation and Terminology

In Bayesian terminology:

$P(B_j)$ is the prior probability of hypothesis $B_j$ (before observing data)
$P(A|B_j)$ is the likelihood of observing data $A$ given hypothesis $B_j$
$P(B_j|A)$ is the posterior probability of hypothesis $B_j$ (after observing data)
$\sum_i P(A|B_i)P(B_i)$ is the marginal likelihood or evidence

The theorem states: Posterior is proportional to likelihood times prior.

Example

Medical Diagnosis: A rare disease affects 0.1% of the population. A diagnostic test has 99% sensitivity (detects disease when present) and 95% specificity (correct when disease absent).

Let $D$ = "has disease", $T$ = "tests positive". Given:

Prior: $P(D) = 0.001$ , $P(D^c) = 0.999$
Likelihood: $P(T|D) = 0.99$ , $P(T|D^c) = 0.05$

By Bayes' theorem: $P(D|T) = \frac{P(T|D)P(D)}{P(T|D)P(D) + P(T|D^c)P(D^c)} = \frac{0.99 \times 0.001}{0.99 \times 0.001 + 0.05 \times 0.999}$ $= \frac{0.00099}{0.05094} \approx 0.0194 \approx 2\%$

Despite the test's high accuracy, a positive result only indicates 2% probability of disease due to the low base rate (prior).

Continuous Version

For continuous random variables with densities, Bayes' theorem takes the form: $f_{\Theta|X}(\theta|x) = \frac{f_{X|\Theta}(x|\theta) f_{\Theta}(\theta)}{f_X(x)} = \frac{f_{X|\Theta}(x|\theta) f_{\Theta}(\theta)}{\int f_{X|\Theta}(x|t) f_{\Theta}(t) \, dt}$

This form is fundamental in Bayesian statistics, where we update our beliefs about parameters $\theta$ based on observed data $x$ .

Remark

Bayes' theorem is not controversial—it follows directly from the definition of conditional probability. The philosophical debate in statistics centers on whether we should assign probability distributions to parameters (Bayesian approach) or treat them as fixed unknowns (frequentist approach).