Joint and Conditional Distributions - Key Properties

Understanding independence and covariance structure reveals how random variables relate to each other.

Independence

Definition

Random variables $X$ and $Y$ are independent if: $p_{X,Y}(x,y) = p_X(x) \cdot p_Y(y) \quad \text{(discrete)}$ $f_{X,Y}(x,y) = f_X(x) \cdot f_Y(y) \quad \text{(continuous)}$

Equivalently: $P(X \in A, Y \in B) = P(X \in A)P(Y \in B)$ for all sets $A, B$ .

Test for Independence: The joint factors as a product of marginals.

Example

If $f_{X,Y}(x,y) = 6xy$ for $0 < x < 1, 0 < y < 1$ :

Marginals: $f_X(x) = \int_0^1 6xy \, dy = 3x$ , $f_Y(y) = \int_0^1 6xy \, dx = 3y$

Product: $f_X(x) \cdot f_Y(y) = 9xy \neq 6xy$

Not independent! (But if it were $9xy$ , they'd be independent.)

Covariance and Correlation

Definition

The covariance is: $\text{Cov}(X,Y) = E[(X-\mu_X)(Y-\mu_Y)] = E[XY] - E[X]E[Y]$

The correlation is: $\rho(X,Y) = \frac{\text{Cov}(X,Y)}{\sigma_X \sigma_Y}$

Properties:

If $X, Y$ independent → $\text{Cov}(X,Y) = 0$ (converse not true!)
$-1 \leq \rho \leq 1$
$|\rho| = 1$ iff $Y = aX + b$ (perfect linear relationship)
$\text{Var}(X + Y) = \text{Var}(X) + \text{Var}(Y) + 2\text{Cov}(X,Y)$

Example

Let $X \sim \mathcal{N}(0,1)$ and $Y = X^2$ . Then:

$E[XY] = E[X^3] = 0$ (odd function)
$E[X] = 0$ , $E[Y] = 1$
$\text{Cov}(X,Y) = 0 - 0 \cdot 1 = 0$

Yet $X$ and $Y$ are clearly dependent! Zero covariance doesn't imply independence.

Bivariate Normal Distribution

Definition

$(X,Y)$ have bivariate normal distribution if their joint PDF is: $f(x,y) = \frac{1}{2\pi\sigma_X\sigma_Y\sqrt{1-\rho^2}} \exp\left(-\frac{Q}{2(1-\rho^2)}\right)$

where: $Q = \left(\frac{x-\mu_X}{\sigma_X}\right)^2 - 2\rho\frac{(x-\mu_X)(y-\mu_Y)}{\sigma_X\sigma_Y} + \left(\frac{y-\mu_Y}{\sigma_Y}\right)^2$

Parameters: $\mu_X, \mu_Y, \sigma_X, \sigma_Y, \rho$

Key Property: For bivariate normal, uncorrelated ( $\rho = 0$ ) implies independent!

Conditional Distribution: $X|Y=y \sim \mathcal{N}\left(\mu_X + \rho\frac{\sigma_X}{\sigma_Y}(y-\mu_Y), \sigma_X^2(1-\rho^2)\right)$

This is the foundation of linear regression.

Remark

Independence is stronger than zero covariance. While independent random variables always have zero covariance, the converse requires special structure (e.g., bivariate normality). Understanding this distinction is crucial for proper statistical modeling.