Lagrange Multipliers

Lagrange multipliers provide a method for finding extrema of a function subject to constraints. The key idea: at a constrained extremum, the gradient of the objective function is parallel to the gradient of the constraint. This reduces constrained optimization to solving a system of equations. Lagrange multipliers are ubiquitous in economics, physics, and machine learning.

Statement

Theorem10.1Lagrange Multipliers

To find the extrema of $f(\mathbf{x})$ subject to the constraint $g(\mathbf{x}) = 0$ , solve the system

$\nabla f = \lambda \nabla g, \quad g(\mathbf{x}) = 0$

for $\mathbf{x}$ and $\lambda$ (the Lagrange multiplier). Solutions are candidates for constrained extrema.

RemarkGeometric interpretation

At a constrained extremum, $\nabla f$ must be perpendicular to the constraint manifold (otherwise, moving along the manifold would increase or decrease $f$ ). Since $\nabla g$ is also perpendicular to the manifold, $\nabla f$ and $\nabla g$ are parallel: $\nabla f = \lambda \nabla g$ .

Examples

ExampleMaximize xy subject to x² + y² = 1

Objective: $f(x, y) = xy$ . Constraint: $g(x, y) = x^2 + y^2 - 1 = 0$ .

Lagrange condition: $\nabla f = \lambda \nabla g$ gives $(y, x) = \lambda (2x, 2y)$ , so $y = 2\lambda x$ and $x = 2\lambda y$ . Thus $x = 4\lambda^2 x$ , so either $x = 0$ or $\lambda^2 = 1/4$ .

If $\lambda = 1/2$ : $y = x$ , so $2x^2 = 1$ , giving $(x, y) = (\pm 1/\sqrt{2}, \pm 1/\sqrt{2})$ with $f = 1/2$ (maximum).
If $\lambda = -1/2$ : $y = -x$ , giving $f = -1/2$ (minimum).

ExampleDistance from origin to surface

Minimize $f(x, y, z) = x^2 + y^2 + z^2$ subject to $g(x, y, z) = x^2 + 2y^2 + 3z^2 - 1 = 0$ (ellipsoid). Lagrange: $(2x, 2y, 2z) = \lambda (2x, 4y, 6z)$ . This gives $x = \lambda x$ , $y = 2\lambda y$ , $z = 3\lambda z$ . Solutions occur when one variable is nonzero and others vanish, giving minimum distance along principal axes.

Multiple constraints

Theorem10.2Multiple constraints

To optimize $f$ subject to $g_1 = 0, \ldots, g_k = 0$ , solve

$\nabla f = \lambda_1 \nabla g_1 + \cdots + \lambda_k \nabla g_k, \quad g_i = 0 \text{ for all } i.$

ExampleTwo constraints

Minimize $f(x, y, z) = x^2 + y^2 + z^2$ subject to $x + y + z = 1$ and $x^2 + y^2 = 1$ . Lagrange: $\nabla f = \lambda_1 (1, 1, 1) + \lambda_2 (2x, 2y, 0)$ . Solving gives critical points.

Justification via Implicit Function Theorem

RemarkWhy Lagrange multipliers work

The Implicit Function Theorem guarantees that the constraint $g(\mathbf{x}) = 0$ locally defines a manifold $M$ . To optimize $f$ on $M$ , we need $\nabla f|_M = 0$ (the gradient restricted to $M$ ). Since directions tangent to $M$ are perpendicular to $\nabla g$ , the condition $\nabla f = \lambda \nabla g$ ensures $\nabla f|_M = 0$ .

Summary

Lagrange multipliers for constrained optimization:

Solve $\nabla f = \lambda \nabla g$ with $g = 0$ .
Geometric: $\nabla f$ parallel to $\nabla g$ at extrema.
Multiple constraints: $\nabla f = \sum \lambda_i \nabla g_i$ .
Justification: Implicit Function Theorem.

See Extrema and Implicit Function Theorem.