Without loss of generality, assume $\mathbf{x}_0 = \mathbf{0}$. Given $\varepsilon > 0$ small enough that $\overline{B_\varepsilon} = \{\|\mathbf{x}\| \leq \varepsilon\} \subset D$.

Since $V$ is continuous, positive definite, and $V(\mathbf{0}) = 0$, the minimum value on the sphere $\|\mathbf{x}\| = \varepsilon$ satisfies:

$m = \min_{\|\mathbf{x}\| = \varepsilon} V(\mathbf{x}) > 0.$

By continuity of $V$ at $\mathbf{0}$, there exists $\delta > 0$ (with $\delta < \varepsilon$) such that:

$\|\mathbf{x}\| < \delta \implies V(\mathbf{x}) < m.$

Now let $\mathbf{x}(t)$ be a solution with $\|\mathbf{x}(0)\| < \delta$. We claim $\|\mathbf{x}(t)\| < \varepsilon$ for all $t \geq 0$.

Since $\dot{V}(\mathbf{x}) = \nabla V(\mathbf{x}) \cdot \mathbf{F}(\mathbf{x}) \leq 0$, the function $t \mapsto V(\mathbf{x}(t))$ is non-increasing. Therefore:

$V(\mathbf{x}(t)) \leq V(\mathbf{x}(0)) < m \quad \text{for all } t \geq 0.$

Suppose for contradiction that there exists $t^* > 0$ with $\|\mathbf{x}(t^*)\| = \varepsilon$. Then:

$V(\mathbf{x}(t^*)) \geq m,$

contradicting $V(\mathbf{x}(t^*)) < m$. Hence $\|\mathbf{x}(t)\| < \varepsilon$ for all $t \geq 0$, proving stability. $\blacksquare$

From Part 1, the origin is stable. Choose $\varepsilon > 0$ and $\delta > 0$ as above so that $\|\mathbf{x}(0)\| < \delta$ implies $\|\mathbf{x}(t)\| < \varepsilon$ for all $t \geq 0$.

We must show $\mathbf{x}(t) \to \mathbf{0}$ as $t \to \infty$. Since $V(\mathbf{x}(t))$ is non-increasing and bounded below by $0$, it converges:

$\lim_{t \to \infty} V(\mathbf{x}(t)) = L \geq 0.$

We claim $L = 0$. Suppose $L > 0$. Then for all $t \geq 0$, we have $V(\mathbf{x}(t)) \geq L > 0$, which means $\mathbf{x}(t)$ stays outside some neighborhood of $\mathbf{0}$. Specifically, there exists $r > 0$ such that $\|\mathbf{x}(t)\| \geq r$ for all $t \geq 0$ (since $V$ is positive definite).

On the compact set $\{r \leq \|\mathbf{x}\| \leq \varepsilon\}$, since $\dot{V}$ is negative definite (i.e., $\dot{V}(\mathbf{x}) < 0$ for $\mathbf{x} \neq \mathbf{0}$), there exists $\alpha > 0$ such that:

$\dot{V}(\mathbf{x}) \leq -\alpha \quad \text{for all } r \leq \|\mathbf{x}\| \leq \varepsilon.$

Therefore:

$V(\mathbf{x}(t)) = V(\mathbf{x}(0)) + \int_0^t \dot{V}(\mathbf{x}(s))\,ds \leq V(\mathbf{x}(0)) - \alpha t.$

As $t \to \infty$, the right side tends to $-\infty$, contradicting $V \geq 0$. Hence $L = 0$, and since $V$ is positive definite, this forces $\mathbf{x}(t) \to \mathbf{0}$. $\blacksquare$

Let $U = \{\mathbf{x} \in D : V(\mathbf{x}) > 0\}$. By hypothesis, $\mathbf{0}$ is a limit point of $U$ (every neighborhood of $\mathbf{0}$ contains points where $V > 0$), and $\dot{V}(\mathbf{x}) > 0$ on $U$.

Choose $\varepsilon > 0$ with $\overline{B_\varepsilon} \subset D$, and let $U_\varepsilon = U \cap B_\varepsilon$. Pick $\mathbf{x}_0 \in U_\varepsilon$, so $V(\mathbf{x}_0) > 0$.

Along the trajectory $\mathbf{x}(t)$ starting at $\mathbf{x}_0$, since $\dot{V} > 0$ on $U$:

$V(\mathbf{x}(t)) > V(\mathbf{x}_0) > 0 \quad \text{as long as } \mathbf{x}(t) \in U_\varepsilon.$

In particular, $\mathbf{x}(t)$ cannot approach $\mathbf{0}$ (where $V = 0$). Since $V(\mathbf{x}(t))$ is strictly increasing and bounded on $\overline{B_\varepsilon}$, the trajectory must eventually leave $B_\varepsilon$.

More precisely, if $\mathbf{x}(t)$ remained in $U_\varepsilon$ for all $t \geq 0$, then $V(\mathbf{x}(t)) \to L > V(\mathbf{x}_0) > 0$. The $\omega$-limit set would be nonempty (contained in $\overline{B_\varepsilon}$), invariant, and satisfy $\dot{V} = 0$ there (since $V$ is constant on $\omega$-limit sets when $V$ is monotone). But $\dot{V} > 0$ on $U$, so the $\omega$-limit set would lie outside $U$, contradicting $V \geq L > 0$ there. Hence $\mathbf{x}(t)$ must leave $B_\varepsilon$ in finite time.

Since this holds for arbitrarily small $\varepsilon$, the equilibrium is unstable. $\blacksquare$