Goal: Show that if $D^2 > 0$ and $D \cdot H = 0$, we reach a contradiction.

Assume $D$ is a divisor with $D \cdot H = 0$ and $D^2 > 0$. For any integer $n$, consider the divisor $nD$. By the Riemann--Roch theorem for surfaces:

$\chi(\mathcal{O}_X(nD)) = \frac{1}{2}(nD) \cdot (nD - K_X) + \chi(\mathcal{O}_X) = \frac{n^2}{2}D^2 - \frac{n}{2}D \cdot K_X + \chi(\mathcal{O}_X).$

Since $D^2 > 0$, the leading term $\frac{n^2}{2}D^2$ dominates for $|n|$ large. Therefore:

$\chi(\mathcal{O}_X(nD)) \to +\infty \quad \text{as } |n| \to \infty.$

Now recall that $\chi(\mathcal{O}_X(nD)) = h^0(nD) - h^1(nD) + h^2(nD)$. Since $h^1(nD) \geq 0$, we have:

$h^0(nD) + h^2(nD) \geq \chi(\mathcal{O}_X(nD)) \to +\infty.$

So at least one of $h^0(nD)$ or $h^2(nD)$ must grow without bound as $|n| \to \infty$.

By Serre duality, $h^2(\mathcal{O}_X(nD)) = h^0(\mathcal{O}_X(K_X - nD))$. So the conclusion of Step 1 says:

For $|n|$ large, either $h^0(nD) \to \infty$ or $h^0(K_X - nD) \to \infty$.

Case A: $h^0(nD) \geq 1$ for some $n > 0$. Then $nD$ is linearly equivalent to an effective divisor, say $nD \sim E \geq 0$. Since $H$ is ample and $E$ is effective, we have $E \cdot H > 0$ (an ample divisor has strictly positive intersection with every effective divisor). But $E \cdot H = nD \cdot H = n(D \cdot H) = 0$, a contradiction.

Case B: $h^0(K_X - nD) \geq 1$ for some $n > 0$. Then $K_X - nD \sim F \geq 0$, so $F \cdot H > 0$ by ampleness. But $F \cdot H = (K_X - nD) \cdot H = K_X \cdot H - n(D \cdot H) = K_X \cdot H$. This is a fixed quantity (independent of $n$), so this does not yet give a contradiction for a single $n$.

However, we can be more precise. Since $h^0(nD) + h^0(K_X - nD) \geq \chi(\mathcal{O}_X(nD)) \to +\infty$, and since $h^0(nD) = 0$ for all large $n$ (by Case A, any nonzero $h^0(nD)$ leads to contradiction), we must have $h^0(K_X - nD) \to \infty$. In particular, $h^0(K_X - nD) \geq 1$ for all sufficiently large $n$.

Now apply the same argument to $-D$: since $D \cdot H = 0$ implies $(-D) \cdot H = 0$, and $(-D)^2 = D^2 > 0$, replace $D$ by $-D$. Then the same reasoning gives $h^0(-nD) = 0$ for all $n > 0$ (by Case A for $-D$), and $h^0(K_X + nD) \to \infty$.

In particular, for $n$ large enough: $K_X - nD \sim F_n \geq 0$ and $K_X + nD \sim G_n \geq 0$. Adding: $2K_X \sim F_n + G_n \geq 0$, which is independent of $n$. But $F_n \cdot H = K_X \cdot H - nD \cdot H = K_X \cdot H$ and $G_n \cdot H = K_X \cdot H + nD \cdot H = K_X \cdot H$. So the effective decomposition $F_n + G_n \sim 2K_X$ has bounded degree with respect to $H$, yet $F_n$ and $G_n$ are both effective with $F_n + G_n$ fixed. Since effective divisors of bounded degree on a projective surface form a bounded family, there are only finitely many possibilities. But this must hold for infinitely many $n$, so $F_n = F_m$ and $G_n = G_m$ for some $n \neq m$. Then $(F_n - F_m) \sim (n - m)D \sim 0$ as effective divisors, which implies $nD - mD$ is principal, hence $(n-m)D \sim 0$ with $n \neq m$, contradicting $D^2 > 0$.

This completes the proof that $D^2 > 0$ is impossible.

Now suppose $D \cdot H = 0$ and $D^2 = 0$, and $D \not\equiv 0$. We derive a contradiction.

Since $D \not\equiv 0$, there exists a divisor $C$ with $D \cdot C \neq 0$. We may assume $D \cdot C > 0$ (replacing $D$ by $-D$ if necessary, which preserves $D \cdot H = 0$ and $D^2 = 0$).

For integers $m, n$, consider the divisor $mH + nD$. Its self-intersection is:

$(mH + nD)^2 = m^2 H^2 + 2mn(H \cdot D) + n^2 D^2 = m^2 H^2$

since $H \cdot D = 0$ and $D^2 = 0$. Also:

$(mH + nD) \cdot H = m H^2 + n(D \cdot H) = mH^2.$

So for $m > 0$, the divisor $mH + nD$ has positive self-intersection $m^2 H^2 > 0$ and positive intersection $mH^2 > 0$ with the ample class $H$. By Riemann--Roch:

$\chi(\mathcal{O}(mH + nD)) = \frac{m^2 H^2}{2} - \frac{m(H \cdot K_X) + n(D \cdot K_X)}{2} + \chi(\mathcal{O}_X).$

For $m$ large and fixed $n$, by Serre's vanishing theorem (or asymptotic Riemann--Roch), $h^0(mH + nD) > 0$ for $m \gg 0$. So $mH + nD$ is linearly equivalent to an effective divisor for $m$ sufficiently large (depending on $n$).

Write $mH + nD \sim E_{m,n} \geq 0$. Then:

$E_{m,n} \cdot C = m(H \cdot C) + n(D \cdot C).$

Since $H$ is ample and $C$ is a divisor, $H \cdot C$ is some fixed integer. By choosing $n$ negative with $|n|$ large and $m$ appropriately large (to ensure effectivity), we can make:

$E_{m,n} \cdot C = m(H \cdot C) + n(D \cdot C) < 0.$

But $E_{m,n}$ is effective, and if $C$ were also effective and ample, this would give a contradiction. More carefully: replace $C$ by a large multiple of $H$ plus a correction. Since $D \cdot C > 0$ and we can choose $n < 0$ with $|n|$ as large as we like (adjusting $m$ accordingly), the intersection $E_{m,n} \cdot C$ can be made arbitrarily negative.

However, we can argue more directly. Apply the result from Steps 1--2 to the divisor $D' = mH + nD$ for a suitable pair $(m, n)$. Specifically, choose $n \neq 0$ and set $D' = nD$. Then $D' \cdot H = 0$ and $(D')^2 = n^2 D^2 = 0$. By Riemann--Roch:

$\chi(\mathcal{O}(nD)) = -\frac{n}{2} D \cdot K_X + \chi(\mathcal{O}_X).$

This is linear in $n$. If $D \cdot K_X \neq 0$, then for $n \to +\infty$ or $n \to -\infty$, $\chi(\mathcal{O}(nD)) \to +\infty$, and the same argument as in Step 2 produces a contradiction. If $D \cdot K_X = 0$, then $\chi(\mathcal{O}(nD)) = \chi(\mathcal{O}_X)$ for all $n$, and we use a refined argument with $mH + nD$ as above.

For the refined argument with $D \cdot K_X = 0$: consider $D' = H + tD$ for large integer $t$. Then $D' \cdot H = H^2 > 0$, $(D')^2 = H^2 > 0$, and for $t$ large, $D' \cdot C = H \cdot C + t(D \cdot C)$. Since $D \cdot C > 0$, the divisor $D'$ behaves like an ample class. Apply the Hodge Index Theorem (the part we already proved: the strict inequality $D^2 \leq 0$ for $D \cdot H = 0$) to the divisor $D$ with respect to the "pseudo-ample" class $D'$. Formally: if $E \cdot D' = 0$ and $(D')^2 > 0$, then $E^2 \leq 0$. Taking $E = aD' - bD$ for appropriate $a, b$ to ensure $E \cdot D' = 0$ leads to $E^2 \leq 0$, which forces $D^2 \leq 0$. Since we assumed $D^2 = 0$, this is consistent but shows $D$ must be numerically proportional to any class it pairs nontrivially with, eventually forcing $D \equiv 0$.

A cleaner approach: by the already-proved inequality $D^2 \leq 0$, the form restricted to $H^\perp$ is negative semi-definite. If $D^2 = 0$ and $D \cdot H = 0$, then for any divisor $C$, consider $D' = C - \frac{C \cdot H}{H^2} H$. Then $D' \cdot H = 0$, so $(D')^2 \leq 0$, i.e., $C^2 - \frac{(C \cdot H)^2}{H^2} \leq 0$. Now $D \cdot C = D \cdot D' + D \cdot \frac{C \cdot H}{H^2} H = D \cdot D'$. By the Cauchy--Schwarz inequality for the negative semi-definite form on $H^\perp$:

$(D \cdot D')^2 \geq (-D^2)(-D'^2) \quad \text{(with reversed sign, since the form is negative semi-definite)}.$

Wait -- for a negative semi-definite form $q$, the Cauchy--Schwarz inequality gives $q(x,y)^2 \leq q(x,x) \cdot q(y,y)$, i.e., $(D \cdot D')^2 \leq D^2 \cdot (D')^2$. But $D^2 = 0$, so $(D \cdot D')^2 \leq 0$, hence $D \cdot D' = 0$, hence $D \cdot C = 0$ for all $C$. This means $D \equiv 0$. $\blacksquare$

Theorem. Let $X$ be a smooth projective surface, $H$ an ample divisor. If $D \cdot H = 0$, then $D^2 \leq 0$, with equality iff $D \equiv 0$.

Proof. Suppose $D \cdot H = 0$.

(i) Proving $D^2 \leq 0$. Assume for contradiction that $D^2 > 0$. By Riemann--Roch:

$\chi(\mathcal{O}_X(nD)) = \frac{n^2}{2} D^2 - \frac{n}{2} D \cdot K_X + \chi(\mathcal{O}_X) \to +\infty$

as $|n| \to \infty$, since the $n^2$ term dominates. So $h^0(nD) + h^0(K_X - nD) \geq \chi(\mathcal{O}_X(nD)) \to +\infty$.

If $h^0(nD) \geq 1$ for some $n > 0$, then $nD$ is effective, so $nD \cdot H > 0$ by ampleness of $H$. But $nD \cdot H = 0$, contradiction.

If $h^0(-nD) \geq 1$ for some $n > 0$, then $-nD$ is effective, so $(-nD) \cdot H > 0$, giving $-n(D \cdot H) > 0$, i.e., $0 > 0$, contradiction.

Therefore $h^0(nD) = 0$ for all $n \neq 0$. Then $h^0(K_X - nD) \to \infty$, so $K_X - nD$ is effective for large $n > 0$, and $K_X + nD$ is effective for large $n > 0$. Adding: $F_n + G_n \sim 2K_X$ with $F_n = K_X - nD \geq 0$ and $G_n = K_X + nD \geq 0$. Since the set of effective decompositions of $2K_X$ is finite (bounded family), for large $n, m$ with $n \neq m$: $F_n = F_m$, giving $(n - m)D \sim 0$, so $D^2 = 0$, contradicting $D^2 > 0$.

(ii) Proving equality implies $D \equiv 0$. Suppose $D \cdot H = 0$ and $D^2 = 0$. By part (i), the intersection form is negative semi-definite on $H^\perp$. For any divisor $C$, set $C' = C - \frac{C \cdot H}{H^2} H \in H^\perp \otimes \mathbb{Q}$. Then $D \cdot C = D \cdot C'$ (since $D \cdot H = 0$). By Cauchy--Schwarz for the negative semi-definite form:

$(D \cdot C')^2 \leq D^2 \cdot (C')^2 = 0 \cdot (C')^2 = 0.$

Hence $D \cdot C' = 0$, so $D \cdot C = 0$ for all divisors $C$. Therefore $D \equiv 0$. $\blacksquare$

Over $\mathbb{C}$, a more conceptual proof uses the Hodge decomposition.

Setup. Let $X$ be a smooth projective surface over $\mathbb{C}$. The second cohomology decomposes as:

$H^2(X, \mathbb{C}) = H^{2,0}(X) \oplus H^{1,1}(X) \oplus H^{0,2}(X).$

The intersection form is given by the cup product pairing on $H^2(X, \mathbb{Z})$.

The Hodge--Riemann bilinear relations. Let $\omega \in H^{1,1}(X)$ be the class of a Kahler form (i.e., an ample class). Define the primitive cohomology $P^{1,1} = \ker(\omega \wedge -: H^{1,1} \to H^{3,3}) = (\omega)^\perp \cap H^{1,1}$. The Hodge--Riemann relations state:

• The pairing is positive definite on $H^{2,0} \oplus H^{0,2}$ (after adjusting signs).
• The pairing restricted to $\mathbb{R} \cdot \omega$ is positive definite (since $\omega^2 = \int_X \omega \wedge \omega > 0$).
• The pairing restricted to $P^{1,1}_{\mathbb{R}} = \omega^\perp \cap H^{1,1}_{\mathbb{R}}$ is negative definite.

Deducing the algebraic statement. The Neron--Severi group $\operatorname{NS}(X)$ embeds into $H^{1,1}(X) \cap H^2(X, \mathbb{Z})$. The ample class $H$ maps to a Kahler class $\omega$, and $D \cdot H = 0$ in the algebraic sense translates to $[D] \in P^{1,1}$. The negative definiteness of the pairing on $P^{1,1}$ gives $D^2 < 0$ for $[D] \neq 0$, and $D^2 = 0$ iff $[D] = 0$ in $H^{1,1}$.

Since $\operatorname{NS}(X)$ injects into $H^{1,1}$ modulo torsion, $[D] = 0$ in $H^{1,1}$ iff $D \equiv 0$ numerically.

If $D^2 \leq 0$ or $C^2 \leq 0$, the right side is $\leq 0$ and the inequality is trivially satisfied (since $(C \cdot D)^2 \geq 0$).

Suppose $C^2 > 0$. By Nakai--Moishezon, if also $C \cdot E > 0$ for all curves $E$, then $C$ is ample and we can use $C$ as the ample class. Set $D' = D - \frac{D \cdot C}{C^2} C$. Then:

$D' \cdot C = D \cdot C - \frac{D \cdot C}{C^2} \cdot C^2 = 0.$

By the Hodge Index Theorem applied with the ample class replaced by a suitable positive multiple of $C$:

$D'^2 \leq 0.$

Expanding: $D'^2 = D^2 - 2 \frac{D \cdot C}{C^2}(D \cdot C) + \frac{(D \cdot C)^2}{(C^2)^2} C^2 = D^2 - \frac{(D \cdot C)^2}{C^2} \leq 0.$

Rearranging: $(C \cdot D)^2 \geq C^2 \cdot D^2$. Equality holds iff $D' \equiv 0$, iff $D \equiv \frac{D \cdot C}{C^2} C$, iff $C$ and $D$ are numerically proportional. $\blacksquare$

Let $H_Y$ be an ample divisor on $Y$, and let $H = f^* H_Y$. Then $H$ is nef and big on $X$ (it is the pullback of an ample divisor), and $H \cdot E_i = H_Y \cdot f_* E_i = 0$ for each $i$ (since $f_* E_i = 0$ as a cycle on $Y$, because $E_i$ is contracted by $f$).

Any divisor $D = \sum a_i E_i$ supported on the exceptional locus satisfies $D \cdot H = \sum a_i (E_i \cdot H) = 0$. By the Hodge Index Theorem (extended to nef and big classes by a limit argument, or by perturbing $H$ to a nearby ample class $H + \epsilon A$):

$D^2 \leq 0, \quad \text{with equality iff } D \equiv 0.$

But on a smooth surface, $D = \sum a_i E_i \equiv 0$ with all $E_i$ exceptional implies all $a_i = 0$ (since the $E_i$ are linearly independent in $\operatorname{NS}(X)$). Therefore the quadratic form $D \mapsto D^2$ is strictly negative definite on $\bigoplus \mathbb{Z} E_i$. $\blacksquare$