Hypothesis Testing Framework

Hypothesis testing provides a formal framework for making decisions about population parameters based on sample data, controlling the probability of erroneous conclusions.

Basic Concepts

Definition

A hypothesis test involves:

Null hypothesis $H_0$ : the default assumption (e.g., $\theta = \theta_0$ )
Alternative hypothesis $H_1$ (or $H_a$ ): the claim to be supported (e.g., $\theta \neq \theta_0$ , $\theta > \theta_0$ , or $\theta < \theta_0$ )
Test statistic $T = T(X_1, \ldots, X_n)$ : a function of the data
Rejection region $R$ : the set of values of $T$ that lead to rejecting $H_0$

Definition

The two types of errors are:

Type I error: Rejecting $H_0$ when it is true. The probability $\alpha = P(\text{reject } H_0 | H_0 \text{ true})$ is the significance level.
Type II error: Failing to reject $H_0$ when it is false. The probability $\beta = P(\text{fail to reject } H_0 | H_1 \text{ true})$ .
Power: $1 - \beta = P(\text{reject } H_0 | H_1 \text{ true})$ measures the ability to detect a true effect.

The Testing Procedure

ExampleZ-test for the mean

Test $H_0: \mu = \mu_0$ vs. $H_1: \mu \neq \mu_0$ with known $\sigma$ at level $\alpha$ :

Test statistic: $Z = \frac{\bar{X} - \mu_0}{\sigma/\sqrt{n}}$
Under $H_0$ : $Z \sim N(0,1)$
Rejection region: $|Z| > z_{\alpha/2}$

For $\alpha = 0.05$ : reject if $|Z| > 1.96$ . The type II error depends on the true $\mu$ : $\beta(\mu) = \Phi\left(z_{\alpha/2} - \frac{\mu - \mu_0}{\sigma/\sqrt{n}}\right) - \Phi\left(-z_{\alpha/2} - \frac{\mu - \mu_0}{\sigma/\sqrt{n}}\right)$ .

P-Values

RemarkP-value interpretation

The $p$ -value is the probability, under $H_0$ , of observing a test statistic as extreme as or more extreme than the one actually observed. We reject $H_0$ when $p < \alpha$ . The $p$ -value quantifies the strength of evidence against $H_0$ : smaller $p$ means stronger evidence. However, the $p$ -value is NOT the probability that $H_0$ is true, a common misinterpretation.