Power Analysis and Sample Size

Power analysis determines the sample size needed to detect an effect of a given size with a specified probability, balancing the tradeoff between type I and type II errors.

The Power Function

Definition

The power function of a test with rejection region $R$ is $\beta(\theta) = P_\theta(T \in R) = P(\text{reject } H_0 \mid \theta)$ This gives the probability of rejection as a function of the true parameter value $\theta$ :

For $\theta \in \Theta_0$ (null): $\beta(\theta) \leq \alpha$ (size constraint)
For $\theta \in \Theta_1$ (alternative): $\beta(\theta) = 1 - \beta_{\text{type II}}(\theta)$ (power)

Definition

The effect size quantifies the magnitude of the departure from $H_0$ :

For testing $\mu = \mu_0$ : Cohen's $d = |\mu - \mu_0|/\sigma$
Small: $d = 0.2$ , Medium: $d = 0.5$ , Large: $d = 0.8$

The effect size, along with $\alpha$ and $n$ , determines the power of the test.

Sample Size Formulas

Theorem9.3Sample Size for Z-test

For a two-sided $Z$ -test of $H_0: \mu = \mu_0$ at level $\alpha$ with power $1 - \beta$ at $\mu = \mu_1$ : $n = \left(\frac{(z_{\alpha/2} + z_\beta)\sigma}{\mu_1 - \mu_0}\right)^2 = \left(\frac{z_{\alpha/2} + z_\beta}{d}\right)^2$ where $d = |\mu_1 - \mu_0|/\sigma$ is the effect size.

ExampleRequired sample size

To detect an effect size $d = 0.5$ with $\alpha = 0.05$ (two-sided) and power $0.80$ ( $z_{0.025} = 1.96$ , $z_{0.20} = 0.84$ ): $n = \left(\frac{1.96 + 0.84}{0.5}\right)^2 = \left(\frac{2.80}{0.5}\right)^2 = 31.36 \implies n = 32$ For a small effect $d = 0.2$ : $n = (2.80/0.2)^2 = 196$ .

Multiple Testing

RemarkThe multiple comparisons problem

When conducting $m$ simultaneous tests at level $\alpha$ , the probability of at least one false positive is $1 - (1-\alpha)^m$ , which grows rapidly with $m$ . The Bonferroni correction tests each at level $\alpha/m$ , controlling the family-wise error rate (FWER). For large $m$ (e.g., genomics), the Benjamini-Hochberg procedure controls the false discovery rate (FDR) $= E[\text{false positives}/\text{total positives}]$ , which is less conservative and more powerful.