Geodesic Convexity Theory for the Induced-Metric Optimizer

1. Setup

A function $L: \mathbb{R}^N \to \mathbb{R}$ is geodesically convex with respect to a Riemannian metric $g$ if its Riemannian Hessian is positive semidefinite at every point. A geodesic in this context is the shortest path between two points measured using $g$, equivalently a curve whose tangent vector is parallel-transported along itself. Geodesically convex functions are the Riemannian analogue of convex functions: they have no saddle points and no spurious local minima, and every local minimum is the global minimum. This entry asks when the induced metric from a loss embedding can make a non-convex loss geodesically convex.

The induced metric arising from the loss embedding $\phi(\theta) = (\theta, L(\theta))$ with ambient metric $\mathrm{diag}(\gamma, \xi)$ has the form

$$g_{ij}(\theta) = \gamma_{ij}(\theta) + \xi\, \ell_i(\theta)\, \ell_j(\theta),$$

where $\ell = \nabla L$ and $\gamma$ is a positive-definite matrix field on parameter space. The Riemannian Hessian of $L$ with respect to $g$ has the closed form

$$(\mathrm{Hess}_g L)_{ij} = \frac{H_{ij} - C_{ij}}{1 + \xi\,\|\ell\|_\gamma^2},$$

where $H_{ij} = \partial_i \partial_j L$ is the ordinary Euclidean Hessian, $\|\ell\|_\gamma^2 = \ell^\top \gamma^{-1} \ell$, and

$$C_{ij} = \Gamma^k_{ij}(\gamma)\,\ell_k$$

is what we call the curvature correction. The full derivation of this formula is in Appendix A, which also defines the Christoffel bracket and Christoffel symbols used throughout. The denominator is strictly positive, so the eigenvalue sign structure of $\mathrm{Hess}_g L$ is determined entirely by $H - C$.

The point of this entry is the structure of $C$. It depends on the spatial derivatives of $\gamma$ and on the gradient $\ell$, but not on $H$. The correction has its own eigenvalue structure, independent of the Hessian. If $\gamma$ can be chosen so that $C$ cancels the negative eigenvalues of $H$, the function becomes geodesically convex.

The three levels of the hierarchy below correspond to three families of induced-metric optimizers. The fixed variant has $\gamma$ constant. The learnable scalar variant has $\gamma = e^{s(\theta)} I$ for a single scalar function $s$. The learnable diagonal variant has $\gamma = \mathrm{diag}(e^{s_1(\theta)}, \ldots, e^{s_N(\theta)})$ for $N$ independent scale functions. Each level adds degrees of freedom to $\gamma$, and we will see exactly what that unlocks.

Sign convention for $s$. Throughout this entry, $s$ parameterises the metric directly: increasing $s_i$ increases the metric weight in direction $i$, and therefore decreases the parameter step in that direction. The learnable diagonal entry parameterises the inverse metric instead ($\gamma^{-1} = \mathrm{diag}(e^{s_i})$), so its $s$ has the opposite sign: increasing $s_i$ in that entry's convention means stronger preconditioning and a larger step in direction $i$. The underlying object (the Newton preconditioner) is the same in both entries; the formulas look opposite because they parameterise opposite things. When transferring a formula across, substitute $s \to -s$.

One note on dimension. In neural-network applications, $N$ (the number of parameters) is huge, typically in the millions or billions. The two-dimensional examples we use below are pedagogical, because two is the smallest dimension in which a saddle point exists and the algebra is fully transparent. The general-$N$ statements appear afterwards.

The Riemannian Hessian is a (0, 2)-tensor and its eigenvalues with respect to $g$ are coordinate-independent quantities, so the analysis below is genuinely geometric and not an artifact of coordinate choice.

2. Level 0: fixed metric

When $\gamma$ is constant, all its derivatives vanish. The full induced metric $g$ still has non-zero Christoffel symbols, because the $\xi \ell \ell^\top$ piece depends on $\theta$, but the calculation in Appendix A gives them in a special form,

$$\Gamma^k_{ij}(g) = \frac{\xi\,H_{ij}\,\ell^k}{1 + \xi\,\|\ell\|^2}.$$

Contracting with $\ell_k$,

$$C_{ij} = \frac{\xi\,\|\ell\|^2}{1 + \xi\,\|\ell\|^2}\,H_{ij}.$$

The correction is proportional to $H$ itself. Substituting into the Riemannian Hessian formula,

$$(\mathrm{Hess}_g L)_{ij} = \frac{1}{1 + \xi\,\|\ell\|^2}\,H_{ij}.$$

The Riemannian Hessian equals the Euclidean Hessian multiplied by a positive scalar in $(0, 1]$. Concretely, if $H$ has eigenvalues $\lambda_1, \ldots, \lambda_N$ with eigenvectors $v_1, \ldots, v_N$, then $\mathrm{Hess}_g L$ has the same eigenvectors with eigenvalues $\lambda_i / (1 + \xi \|\ell\|^2)$. The signs are preserved, the eigenvectors are preserved, and only the magnitudes are uniformly rescaled. A saddle remains a saddle and a bowl remains a bowl. The fixed induced metric provides adaptive curvature damping but zero curvature correction.

The same conclusion holds for the fixed symmetric off-diagonal extension of the ambient metric (constant coupling vectors $a = b$; see the off-diagonal metric variant). The Christoffel bracket factorises the same way and the Riemannian Hessian remains a scalar multiple of $H$. The asymmetric case ($a \ne b$) does not yield a Riemannian metric in the first place and is treated separately in the off-diagonal entry as a non-symmetric preconditioner. The full derivation is in Appendix B.

3. Level 1: scalar learnable

Set $\gamma = e^{s(\theta)} I$ for a single scalar function $s: \mathbb{R}^N \to \mathbb{R}$, the learnable scalar variant. The metric scales every direction by the same factor $e^s$, but the factor varies in space. This is a conformally flat metric. Its Christoffel symbols are the classical conformal expression,

$$\Gamma^k_{ij}(\gamma) = \tfrac{1}{2}\bigl(\partial_i s\,\delta_{jk} + \partial_j s\,\delta_{ik} - \partial_k s\,\delta_{ij}\bigr).$$

The derivation in our general framework is given in Appendix A. Contracting with $\ell_k$,

$$C_{ij} = \tfrac{1}{2}\bigl(\partial_i s\,\ell_j + \partial_j s\,\ell_i\bigr) - \tfrac{1}{2}(\nabla s \cdot \ell)\,\delta_{ij}.$$

The correction is a symmetric rank-2 outer product (the first two terms) plus a scalar multiple of the identity (the third term).

3.1 The trace identity

Take the trace of $C$, that is, sum the diagonal entries. The diagonal of $C$ has $C_{ii} = \tfrac{1}{2}(\partial_i s\,\ell_i + \partial_i s\,\ell_i) - \tfrac{1}{2}(\nabla s\cdot\ell) = \partial_i s\,\ell_i - \tfrac{1}{2}(\nabla s\cdot\ell)$. Summing over $i$,

$$\mathrm{tr}(C) = \sum_i \partial_i s\,\ell_i - \tfrac{N}{2}(\nabla s\cdot\ell) = (\nabla s\cdot\ell) - \tfrac{N}{2}(\nabla s\cdot\ell) = \frac{2 - N}{2}\,(\nabla s \cdot \ell).$$

This is direct bookkeeping from the explicit formula for $C$, without further assumption. It is a standard consequence of conformal geometry: the trace of the correction is fixed by the dimension and by the inner product of $\nabla s$ with the gradient. In two dimensions ($N = 2$), the trace is identically zero, regardless of how cleverly $s$ varies in space. The conformal correction is trace-free in two dimensions. The metric can redistribute eigenvalues but cannot change their sum. We refer to this as the trace constraint, with the understanding that it is a hard algebraic identity in $N = 2$ and not an inequality.

3.2 Consequences in two dimensions

Consider two two-dimensional examples.

For the asymmetric saddle $L = 3x^2 - y^2$ at $(1, 1)$, the Euclidean Hessian is $H = \mathrm{diag}(6, -2)$ with $\mathrm{tr}(H) = 4 > 0$. Since $\mathrm{tr}(H - C) = \mathrm{tr}(H) = 4$, redistributing the eigenvalues to $(2, 2)$ would give $H - C = 2I$, which is positive definite. Choosing $\nabla s = (6/5, 2/5)$ gives exactly $C = \mathrm{diag}(4, -4)$, and the Riemannian Hessian is $2I$ times the positive scalar $1/(1 + \xi \|\ell\|^2_\gamma)$. Geodesic convexity is achieved.

For the symmetric saddle $L = x^2 - y^2$ at $(1, 1)$, the Euclidean Hessian is $H = \mathrm{diag}(2, -2)$ with $\mathrm{tr}(H) = 0$. Since $\mathrm{tr}(H - C) = 0$ and a real symmetric matrix with zero trace must have at least one non-positive eigenvalue, strict geodesic convexity is impossible for the scalar metric here at any choice of $\nabla s$.

So in two dimensions the scalar metric can fix saddles where positive curvature dominates ($\mathrm{tr}(H) > 0$) but cannot fix symmetric or negative-dominant saddles.

3.3 What about $N > 2$?

For $N > 2$ the trace identity becomes $\mathrm{tr}(C) = ((2-N)/2)(\nabla s \cdot \ell)$, which is no longer identically zero (it vanishes only when $\nabla s \cdot \ell = 0$). The hard trace obstruction is gone, and the scalar metric can in principle shift the sum of eigenvalues. The scalar metric is nonetheless still limited in higher dimensions, for a different reason.

The scalar metric has $N$ free parameters (the components $\partial_k s$ for $k = 1, \ldots, N$). The condition that $H - C$ be positive definite involves controlling all $N(N+1)/2$ entries of a symmetric matrix. For $N > 2$, the scalar metric is under-determined: it has fewer knobs than constraints. This is not a proof that it always fails, but it does mean that the scalar class is highly restricted in high dimensions. It can fix specific easy Hessians, those that are a low-rank perturbation away from positive-definite, but not arbitrary ones.

A concrete example. With $H = \mathrm{diag}(1, -1, -1, -1, -1)$ in $N = 5$, numerically optimizing $\lambda_{\min}(H - C)$ over the components of $\nabla s$ using the SDP described in Section 4.4 returns a maximum value of approximately $-0.8$. Geodesic convexity is unattainable here, even though the trace constraint is formally lifted. The full numerical run is in the attached Mathematica notebook.

The general lesson is that the scalar learnable metric works when the Hessian is easy and the negative-eigenvalue subspace has small rank, but it lacks the degrees of freedom to handle arbitrary Hessians in high dimensions.

Since real neural-network applications have $N$ in the millions, the two-dimensional-only trace constraint is not directly the binding obstruction in practice. The under-determination is the real practical limitation, and it gets worse with $N$.

4. Level 2: diagonal learnable

The learnable diagonal variant replaces the single scalar $s(\theta)$ with $N$ independent scale functions,

$$\gamma(\theta) = \mathrm{diag}\bigl(e^{s_1(\theta)},\, e^{s_2(\theta)},\, \ldots,\, e^{s_N(\theta)}\bigr).$$

Each $s_i: \mathbb{R}^N \to \mathbb{R}$ is its own function of all parameters. The correction $C$ now depends on the partial derivatives $\partial_k s_i$, one for each pair $(k, i)$. Collect these into a matrix,

$$\sigma_{ki} := \partial_k s_i, \qquad \sigma \in \mathbb{R}^{N \times N}.$$

This $\sigma$ is the Jacobian of the vector-valued function $s = (s_1, \ldots, s_N): \mathbb{R}^N \to \mathbb{R}^N$. Each column $i$ of $\sigma$ is the gradient $\nabla s_i$. The diagonal entries $\sigma_{ii}$ describe how the $i$-th scale changes in its own coordinate direction, and the off-diagonal entries $\sigma_{ki}$ for $k \ne i$ describe cross-coupling between scales and coordinates.

4.1 Christoffel structure

The Christoffel symbols of the diagonal metric, derived in Appendix C, are non-zero in three configurations:

(a) $\Gamma^i_{ii}(\gamma) = \tfrac{1}{2}\,\partial_i s_i$, the "same-index" piece, with no exponential factor.

(b) $\Gamma^k_{ii}(\gamma) = -\tfrac{1}{2}\,e^{s_i - s_k}\,\partial_k s_i$ for $k \ne i$, the "cross-index" piece, with exponential anisotropy factor $e^{s_i - s_k}$.

(c) $\Gamma^j_{ij}(\gamma) = \tfrac{1}{2}\,\partial_i s_j$ and $\Gamma^i_{ij}(\gamma) = \tfrac{1}{2}\,\partial_j s_i$ for $i \ne j$.

All other components vanish. The factors $e^{s_i - s_k}$ are the key new ingredient. They did not appear in the conformal (scalar) case because there all $s_i$ were equal, making every $e^{s_i - s_k} = 1$.

4.2 Why there is no trace constraint

Compute $\mathrm{tr}(C) = \sum_i C_{ii}$ for the diagonal metric. Expanding $C_{ii} = \sum_k \Gamma^k_{ii}\,\ell_k$ using the cases above,

$$\mathrm{tr}(C) = \sum_i \Bigl[\tfrac{1}{2}\,\partial_i s_i \,\ell_i - \tfrac{1}{2}\sum_{k \ne i}\, e^{s_i - s_k}\,\partial_k s_i\,\ell_k\Bigr].$$

This is a sum of terms with different exponential factors $e^{s_i - s_k}$. When all the $s_i$ are equal (the scalar case), every $e^{s_i - s_k} = 1$ and the sum collapses to the clean identity $((2-N)/2)(\nabla s \cdot \ell)$ of Level 1. A sanity-check derivation of this reduction is given in Appendix C. When the $s_i$ are free to differ, the exponential factors are independent positive numbers and no algebraic identity ties the trace to anything specific. The trace can take any real value.

So when we say "the diagonal metric breaks the trace constraint," we do not mean we have violated a real theorem. We mean that the constraint never existed for this metric class. The Level 1 trace identity is an artifact of insisting all $s_i$ be equal, and relaxing that uniformity removes the identity.

4.3 The Jacobian as the design variable, and the role of the norm bound

Once we move to the diagonal metric, the object we have freedom to choose is the Jacobian $\sigma_{ki} = \partial_k s_i$. The correction $C$ is determined as a linear function of $\sigma$ at any fixed $\theta$ (with $\ell$ and the $s_i$ playing the role of constants in this fixed-point analysis). To talk about what is achievable, we need to specify what range of $\sigma$ we allow.

Pointwise feasibility versus a global metric field. The analysis in this section and the Gershgorin construction in §4.7 treat $\sigma$ as a free matrix at the point $\theta$ under consideration. For $\sigma$ to be the Jacobian of an actual $s: \mathbb R^N \to \mathbb R^N$ on a region, the integrability conditions

$$\partial_l \sigma_{ki} = \partial_k \sigma_{li} \qquad \text{for all } i, k, l$$

(symmetry of the Hessian of each component $s_i$) must hold. The constructions below establish pointwise first-jet feasibility: at each $\theta$ separately, there exists a $\sigma$ that achieves geodesic convexity. Whether those pointwise $\sigma$'s can be stitched together into a smooth global metric field $\gamma(\theta) = \mathrm{diag}(e^{s_i(\theta)})$ is a separate (and nontrivial) question that is open and not addressed here. The online learnable-diagonal optimizer in practice uses an EMA-driven $s$ that is not required to be the gradient of any global potential; the pointwise feasibility analysis is the relevant theoretical object for its local behaviour.

We impose the Frobenius-norm bound

$$\|\sigma\|_F := \sqrt{\sum_{k,i}\sigma_{ki}^2} \le R$$

for a budget $R$. There are three reasons for this choice:

1. Frobenius is the natural Hilbert-Schmidt norm on $\mathbb{R}^{N \times N}$. It treats every entry symmetrically and does not single out any coordinate direction.

2. $\|\sigma\|_F^2 = \sum_{k,i}(\partial_k s_i)^2$ is the total Dirichlet energy of the vector field $s$. It is the standard "rate of change" measure used in calculus of variations and in regularity theory for vector fields.

3. The constraint $\|\sigma\|_F \le R$ is convex and quadratic. Combined with the concave objective $\lambda_{\min}(H - C(\sigma))$ (concave because $\lambda_{\min}$ of an affine matrix function is concave), the result is a convex semidefinite program, solvable by standard interior-point methods. Uniqueness of the optimum is not guaranteed in general (active-set ambiguities at the norm-ball boundary, eigenvalue-multiplicity issues), but the optimal value is unique and the set of optimisers is convex.

Other matrix norms would also work and give qualitatively similar results. The operator norm $\|\sigma\|_2$ is the actual Lipschitz constant of $s$; the bound $\|\sigma\|_2 \le \|\sigma\|_F$ means a Frobenius bound implies a Lipschitz bound. The $\ell_1$ and $\ell_\infty$ norms would give similar sign-flip-vs-budget pictures with different quantitative thresholds. We picked Frobenius because it is the smoothest, the most rotation-invariant, and the cleanest for SDP.

The budget $R$ is not a free knob in deployment. In the online learnable-diagonal optimizer, the metric-learning rate $\mu$ controls how aggressively the metric can change per step. Combined with the parameter learning rate $\eta$ and the rate at which the optimizer moves through parameter space, $\mu$ sets the scale of $\sigma$ that the algorithm can produce. Larger $\mu$ allows larger $\|\sigma\|_F$. So the budget $R$ stands in for "how much the online rule can move the metric within a single optimization step." The exact quantitative relationship between $R$ and $\mu$ depends on the specific rule and on the trajectory, but the qualitative correspondence is direct: doubling $\mu$ roughly doubles the achievable $R$.

4.4 The sign-flip budget on the symmetric saddle

Return to $L = x^2 - y^2$ at $(1, 1)$, with $H = \mathrm{diag}(2, -2)$ and $\ell = (2, -2)$, the case where the scalar metric provably failed. With the diagonal metric, the free variables are the four entries of $\sigma \in \mathbb{R}^{2 \times 2}$.

We numerically solve

$$\max_{\|\sigma\|_F \le R}\;\lambda_{\min}\bigl((H - C(\sigma, \ell))\,/\,(1 + \xi\,\|\ell\|^2_\gamma)\bigr),$$

i.e., the smallest eigenvalue of the actual Riemannian Hessian rather than of the unnormalised $H - C$. With $\xi = 1$ and $\gamma = I$ at this evaluation point, $\|\ell\|^2_\gamma = 8$ and the denominator is $9$. Since $C$ is linear in $\sigma$ and the denominator is constant in $\sigma$, the objective is a concave function of $\sigma$. We use Mathematica's NMinimize with the SDP backend; the notebook is attached as notebooks/sign-flip-sdp.nb.

The numerical results (Riemannian-Hessian eigenvalue, after dividing by $9$):

Budget $R$	Max $\lambda_{\min}$ of Riemannian Hessian	Sign flip
0	$-0.222$ ($= -2/9$)	No (matches Level 0)
1.5	$-0.009$	No
1.6	$\approx 0$	Threshold crossed
2	$+0.055$	Yes
5	$+0.360$	Yes
10	$+0.770$	Yes

At $R \approx 1.6$ the minimum eigenvalue crosses zero. With $R = 5$ the saddle is converted to a comfortable Riemannian bowl. This was impossible for the scalar metric at any budget.

4.5 The anti-correlation structure of the optimal $\sigma$

At the optimum, $\sigma$ has a characteristic shape: its diagonal entries are anti-correlated with $\mathrm{sign}(H_{ii})$. Concretely, $\sigma_{ii} < 0$ in positive-curvature directions (shrink the metric where the loss curves up) and $\sigma_{ii} > 0$ in negative-curvature directions (grow the metric where the loss curves down). The off-diagonal entries of $\sigma$ are small in this two-dimensional example. This anti-correlation is the geometric mechanism that produces non-zero trace in $C$. It also motivates the curvature-aware variant of the learnable diagonal optimizer, which adds a term to the online update rule that drives $\sigma_{ii}$ in this anti-correlated direction.

4.6 Other saddles

The same numerical procedure works on every saddle we tested. For the asymmetric saddle $L = x^2 - 3y^2$, where the scalar metric struggles because $\mathrm{tr}(H) < 0$, the diagonal still flips signs at moderate $R$. For the monkey saddle $L = x^3 - 3xy^2$, with its position-dependent Hessian, sign-flipping still works. For the Rosenbrock function in regions of negative curvature, sign-flipping is achievable. The corresponding budget tables and optimal $\sigma$ values are in notebooks/sign-flip-sdp.nb.

4.7 Argument for general $N$: Gershgorin construction

For general dimension, we can give a constructive sign-flip argument. Choose $\sigma$ purely diagonal,

$$\sigma_{ii} = -2M/\ell_i, \qquad \sigma_{ki} = 0 \text{ for } k \ne i.$$

The algebra in Appendix C gives two consequences. Each diagonal entry of $H - C$ shifts by $+M$. The off-diagonal entries of $H - C$ are unchanged.

By Gershgorin's disc theorem, every eigenvalue of a matrix lies in a disc centred at one of its diagonal entries with radius equal to the corresponding off-diagonal row sum. If $M > \max_i (R_i - H_{ii})$, where $R_i = \sum_{j \ne i} |H_{ij}|$ is the $i$-th off-diagonal row sum, every Gershgorin disc lies strictly in the positive half-plane. Then $H - C \succ 0$, and the sign flip is achieved for arbitrary $N$.

This argument is constructive as a pointwise existence statement (see §4.3 on pointwise feasibility versus global metric fields). It shows existence of a sign-flipping $\sigma$ at each $\theta$ separately, requiring $\ell_i \ne 0$ for every $i$, and degenerating on coordinate axes where some $\ell_i = 0$. The pointwise $\sigma$ produced by the construction does not in general satisfy the integrability conditions $\partial_l \sigma_{ki} = \partial_k \sigma_{li}$, so it cannot in general be integrated to a global $s(\theta)$; the construction proves feasibility of the first jet, not existence of a smooth metric field. The Gershgorin construction has $\|\sigma\|_F^2 = 4M^2 \sum_i 1/\ell_i^2$, which grows without bound as any $\ell_i \to 0$. For an isotropic gradient with $\ell_i \sim \|\ell\|/\sqrt N$, the sum $\sum_i 1/\ell_i^2 \sim N^2/\|\ell\|^2$, so the total Frobenius budget scales as $\|\sigma\|_F \sim 2 M N/\|\ell\|$, that is $O(N/\|\ell\|)$ overall. The budget blows up when needed directions have small gradient components, which is when sign-flipping is structurally hardest.

4.8 Verification

The sign-flip results were verified numerically in Mathematica for $N = 3, 5, 10, 50, 100, 1000$ on diagonal Hessians with random off-diagonal perturbations. The notebook notebooks/sign-flip-sdp.nb contains the full code and the eigenvalue tables.

5. The hierarchy

Level	Metric $\gamma$	Correction $C$	Trace identity	What it can fix
0 (constant)	$\gamma$ constant	$C \propto H$	$\mathrm{tr}(C) \propto \mathrm{tr}(H)$	Nothing; eigenvalue signs preserved
0+ (fixed off-diag, $a = b$)	const + fixed symmetric $a = b$	$C \propto H$	$\mathrm{tr}(C) \propto \mathrm{tr}(H)$	Nothing (symmetric off-diagonal case only; the asymmetric $a \ne b$ case adds a non-scalar correction whose general analysis is open, see Appendix B)
1 (scalar)	$e^s I$	Rank-2 + scalar $\cdot$ $I$	$\mathrm{tr}(C) = \frac{2-N}{2}(\nabla s \cdot \ell)$, identically zero in 2D	2D: only saddles with $\mathrm{tr}(H) > 0$. High-$N$: under-determined
2 (diagonal)	$\mathrm{diag}(e^{s_i})$	$N^2$ free entries (the Jacobian $\sigma$); no algebraic identity	Unrestricted	All saddles in principle, for arbitrary $N$

Each level strictly contains the previous. Setting all $s_i$ equal in the diagonal case recovers scalar, and setting $s$ constant recovers fixed. The hierarchy gives a geometric explanation for the empirical performance ordering of induced-metric variants: the variants with richer curvature-correction capability outperform those without, holding implementation cost constant.

6. The critical-point obstruction

Look again at $C_{ij} = \Gamma^k_{ij}(\gamma)\,\ell_k$. Every term contains a factor of $\ell_k$ for some $k$. At any critical point $\theta^*$ where $\nabla L(\theta^*) = 0$, we have $C(\theta^*) \equiv 0$, so

$$\mathrm{Hess}_g L(\theta^*) = H(\theta^*)$$

regardless of the metric $\gamma$. The proof is one line: $\ell = 0$ kills the contraction, and the denominator equals 1.

6.1 What this rules out

Global geodesic convexity is impossible for any loss function with a saddle point. At the saddle, the Riemannian Hessian equals the indefinite Euclidean Hessian, no matter what metric is chosen. This is a fundamental obstruction. The curvature-correction mechanism is most powerful in the bulk and powerless exactly at saddles, where one might most want it to act.

6.2 What this does not rule out

Three things soften the obstruction.

First, optimizers rarely hit exact critical points. Iterates approach saddles asymptotically but generically never land on them, and stochastic noise further prevents it. The obstruction applies to a measure-zero set.

Second, the correction grows linearly with $\|\ell\|$. Near a saddle $\theta^*$, $\ell \approx H \delta\theta$, so $\|C\| \propto \|\delta\theta\|$. The correction weakens as the optimizer approaches the saddle, but the optimizer's velocity decreases in step. What matters is the ratio.

Third, the useful question becomes basin enlargement, not global convexity. The next section makes "the obstruction applies to a measure-zero set" quantitative: the failure region at correction budget $R$ is contained in tubes of radius $O(1/R)$ around each non-minimum critical point.

6.3 Quantitative obstruction: $\varepsilon$-excised sign flip

Section 6.1 is a measure-zero negative statement. Its quantitative complement, which characterises how fast the obstruction decays as the optimizer moves away from a saddle, is the main result of this subsection. The failure region of the attainable sign flip shrinks linearly with the correction budget.

Define the attainable geodesic-convexity basin at budget $R$,

$$\mathcal{B}_g^{\mathrm{opt}}(R) := \bigl\{\theta : \exists\,\sigma\ \text{with}\ \|\sigma\|_F \le R\ \text{and}\ H(\theta) - C(\sigma,\ell(\theta)) \succ 0\bigr\}.$$

At each critical point $\theta^*_k$ of a Morse loss, let

$$\mu^*_k := |\lambda_{\min}(H(\theta^*_k))|, \qquad \rho^*_k := \max_i\Bigl[\sum_{j \ne i} |H_{ij}(\theta^*_k)| - H_{ii}(\theta^*_k)\Bigr]$$

denote the spectral gap and the Gershgorin deficit at $\theta^*_k$. At a minimum, $\rho^*_k \le 0$; at a saddle or maximum, $\rho^*_k > 0$.

Theorem 1ʹ ($\varepsilon$-excised sign flip). Let $L \in C^3(\Omega)$ be Morse on a bounded open set $\Omega \subset \mathbb{R}^N$ with finite critical set $\mathrm{Crit}(L)$. Then for every correction budget $R > 0$,

$$\mathcal{B}_g^{\mathrm{opt}}(R)^c \cap \Omega \;\subseteq\; \bigcup_{k\,:\,\rho^*_k > 0} \overline{B\bigl(\theta^*_k,\,\varepsilon_k(R)\bigr)},$$

with the explicit estimate

$$\varepsilon_k(R) \;\le\; \frac{2 N \rho^*_k}{\mu^*_k\,R} \;+\; O(R^{-2}) \qquad \text{as } R \to \infty.$$

In particular every $\varepsilon_k(R) \to 0$ as $R \to \infty$. The bound is sharp on the symmetric saddle $L = x^2 - y^2$ in $N = 2$, where $\rho^* = \mu^* = 2$ and $\varepsilon(R) = 4/R$.

The theorem is essentially the Gershgorin construction of Appendix C.2 wired through a Frobenius-budget bookkeeping and a Taylor expansion at the critical point.

Proof. Apply the diagonal-only choice $\sigma_{ii} = -2 M/\ell_i$ from Appendix C.2. The resulting correction is $C = -M\,I$, so $H - C = H + M\,I$ has the same off-diagonals as $H$ and diagonal entries $H_{ii} + M$. By Gershgorin's disc theorem, $H - C \succ 0$ whenever $M > \rho(\theta)$, where

$$\rho(\theta) := \max_i\Bigl[\sum_{j \ne i} |H_{ij}(\theta)| - H_{ii}(\theta)\Bigr]$$

is the local Gershgorin deficit. The Frobenius cost of this choice is

$$\|\sigma\|_F = 2 M\Bigl(\sum_i \ell_i^{-2}\Bigr)^{1/2},$$

so the budget constraint $\|\sigma\|_F \le R$ caps $M \le R/(2\sqrt{\sum_i \ell_i^{-2}})$. The diagonal construction succeeds when both inequalities can be simultaneously satisfied, equivalently when

$$\Bigl(\sum_i \ell_i^{-2}(\theta)\Bigr)^{1/2} \;<\; \frac{R}{2\,\rho(\theta)}. \tag{6.3.1}$$

This is the budget-translated sign-flip criterion.

Near $\theta^*_k$, change coordinates so $H^*_k := H(\theta^*_k) = \mathrm{diag}(\lambda_1, \ldots, \lambda_N)$ with all $\lambda_i \ne 0$ (Morse). Let $\delta := \theta - \theta^*_k$, $r := \|\delta\|$, and $\hat u := \delta/r$. Taylor expansion gives

$$\ell_i(\theta) = \lambda_i u_i r + O(r^2), \qquad \rho(\theta) = \rho^*_k + O(r),$$

so $\sum_i \ell_i^{-2}(\theta) = r^{-2}\,\sum_i (\lambda_i u_i)^{-2}(1 + O(r))$. Substituting into (6.3.1) gives the per-direction bound

$$r \;>\; \varepsilon(R, \hat u) \;:=\; \frac{2 \rho^*_k}{R}\,\sqrt{\sum_i (\lambda_i u_i)^{-2}} \;+\; O(R^{-2}). \tag{6.3.2}$$

The directional factor $\sqrt{\sum_i (\lambda_i u_i)^{-2}}$ diverges only on the coordinate-axis null-cones (when some $u_i \to 0$), precisely the directions where the diagonal-only construction needs the cross-term lift. On those directions, replace the divergent self-derivative $\sigma_{ii} = -2 M/\ell_i$ with a cross-derivative $\sigma_{k(i),\,i} = 2 M/\ell_{k(i)}$ for a donor coordinate $k(i)$ with $\ell_{k(i)} \ne 0$. The lifted construction (Section 4.7 / Appendix C.2 extension) achieves the same shift $C = -M\,I$ and the lifted directional factor satisfies the uniform bound

$$\sqrt{\sum_i (\lambda_i u_i)^{-2}_{\text{lifted}}} \;\le\; \frac{N}{\mu^*_k}$$

since $\max_i u_i^2 \ge 1/N$ and $|\lambda_i| \ge \mu^*_k$. Substituting into (6.3.2) yields $\varepsilon_k(R) \le 2 N \rho^*_k/(\mu^*_k\,R) + O(R^{-2})$. $\square$

What the theorem buys

Three takeaways.

1. The obstruction is a fence, not a wall. Larger budget shrinks the fence linearly. Doubling $R$ halves the failure tube around each saddle. The $R \to \infty$ limit recovers Section 6.1 as a measure-zero residue.

2. The bound is structural, not directly computable on a real network. The constants $\rho^*_k$ and $\mu^*_k$ are local to each critical point, and a real network has unknown saddle locations. What the theorem gives is the shape of the failure region (tubes of radius $O(1/R)$ around the critical set), not its absolute size at a specific training run. Restricting to a bounded sublevel set $\{L \le L_0\}$ is needed to control the $O(R^{-2})$ remainder (which absorbs a $\|D^3 L\|_{L^\infty}$ factor from the Taylor expansion).

3. The theorem is about attainable basin coverage, not about online tracking. It says a good $\sigma$ exists outside the $\varepsilon$-tube. Whether the EMA-based online estimator finds that $\sigma$ in finite time is a separate question that lives in Section 8.1.

The Morse assumption is load-bearing. Degenerate critical points (e.g., the monkey saddle $L = x^3 - 3 x y^2$, where $H^* = 0$ so $\mu^*_k = 0$) make the bound diverge. The right $\varepsilon$-scaling at a degenerate critical point uses the leading nonzero coefficient of the Taylor expansion of $L$ and gives a slower decay than $1/R$; that case is not treated here.

Numerical verification on the symmetric saddle, the asymmetric saddle $L = x^2 - 3 y^2$, and the quartic well of Section 7.1 is in notebooks/epsilon-excised-sign-flip.nb. On the symmetric saddle the predicted $\varepsilon(R) = 4/R$ matches the observed failure radius along the diagonal direction to scan resolution (table reproduced below).

$R$	$4/R$ (predicted)	observed $\varepsilon$
$2$	$2.000$	$2.001$
$5$	$0.800$	$0.801$
$10$	$0.400$	$0.401$
$50$	$0.080$	$0.081$
$100$	$0.040$	$0.041$

7. Basin enlargement

Around a strict local minimum $\theta^*$ with $H(\theta^*) \succ 0$, define two basins:

$$\mathcal{B}_{\mathrm{eucl}} = \{\theta : H(\theta) \succ 0\}, \qquad \mathcal{B}_g^{\mathrm{opt}}(R) = \{\theta : \exists\,\sigma\text{ with }\|\sigma\|_F \le R\text{ and } H(\theta) - C(\sigma, \ell(\theta)) \succ 0\}.$$

Near the minimum, $\ell \approx 0$, so $C \approx 0$ for any $\sigma$ in the budget. In particular $\sigma = 0$ is feasible and gives $H - C = H \succ 0$. Therefore $\mathcal{B}_{\mathrm{eucl}} \subseteq \mathcal{B}_g^{\mathrm{opt}}(R)$ for every $R \ge 0$: any point at which $H$ is already positive-definite is in the attainable geodesic-convexity basin. This inclusion is about the attainable geodesic basin under optimal choice of $\sigma$, not about the basin produced by any particular online rule (an online rule may apply $\sigma \ne 0$ in the Euclidean basin and inadvertently push $H - C$ out of positive-definiteness). The interesting region is outside $\mathcal{B}_{\mathrm{eucl}}$, where $H$ has negative eigenvalues but $\ell$ is non-zero. There $C$ can compensate.

7.1 The quartic well

A vivid example. $L = x^4 + y^4 - 2x^2 - 2y^2$ has four minima at $(\pm 1, \pm 1)$, saddle points at the origin and along the axes, and Hessian $H = \mathrm{diag}(12x^2 - 4, 12y^2 - 4)$. The Euclidean convexity basin around each minimum extends only to $|x|, |y| > 1/\sqrt{3} \approx 0.58$. The four basins are disconnected square regions.

With the diagonal learnable metric at correction budget $R = 5$, the geodesic convexity basin extends almost to the axes, covering most of the domain. The four disconnected Euclidean basins merge into a single large basin that nearly fills the plane. Only small pockets of non-convexity remain near the coordinate axes, consistent with the critical-point obstruction.

The metric is redrawing the map. An optimizer using this metric can start much further from the minimum and still see a convex-looking landscape all the way down. The boundary of $\mathcal{B}_g$ forms a thin layer around each saddle, with thickness $O(1/R)$ in the correction budget (formalised in Theorem 1ʹ). Larger budgets push the boundary closer to the saddle but never reach it.

8. Trajectory-level analysis

The pointwise theory above asks whether at a given $\theta$ the metric can achieve geodesic convexity. An optimizer, however, moves. Three questions extend the theory to trajectories.

8.1 Does the optimal metric vary smoothly?

At each $\theta$, the optimal $\sigma^*$ solves

$$\sigma^*(\theta) = \arg\max_{\|\sigma\|_F \le R} \lambda_{\min}\bigl(H(\theta) - C(\sigma, \ell(\theta))\bigr).$$

This is the same concave semidefinite program as in Section 4.4. Solutions to smoothly parameterised concave SDPs vary upper-hemicontinuously with the parameters; full smoothness can fail not only at eigenvalue-degeneracy points but also when the active set on the norm-ball boundary switches or the optimum set is non-singleton. Adding a small Tikhonov regulariser to the objective (e.g., $+ \epsilon\|\sigma\|_F^2$) restores smoothness by making the optimum unique. An exponential moving average (EMA)-smoothed online estimator of $\sigma^*$ can track the underlying $\sigma^*$ to within the smoothing window, but with no guarantee of pointwise tracking through active-set transitions.

8.2 Are the coupled dynamics stable?

At a strict local minimum $\theta^*$, the $\theta$ dynamics reduce to standard gradient descent (the denominator factor tends to 1), and the $s$ dynamics relax to $s = 0$ (the identity metric) under regularisation. The Jacobian of the joint update at $(\theta^*, s = 0)$ is block-triangular, because the off-diagonal blocks vanish ($\ell = 0$ kills all cross-terms). Eigenvalues of a block-triangular matrix are the eigenvalues of the diagonal blocks. Both blocks contract under standard learning-rate conditions, giving local asymptotic stability of the joint system without any requirement that the metric converge faster than the parameters.

8.3 Where the corrections operate

Near a minimum, corrections are unnecessary because $H \succ 0$ already; the metric correctly relaxes to identity and composition is trivially coherent.

In the non-convex bulk, the gradient is large, $C$ is strong, regularisation bounds the tracking error, and the denominator damps any aggressive deviation. Corrections compose coherently.

Near a saddle, the gradient is small, the correction weakens by the critical-point obstruction, and the geodesic convexity margin shrinks toward zero. The mechanism hands off to stochastic gradient noise for saddle escape.

The curvature correction is a bulk phenomenon, not a saddle-escape mechanism. It complements stochastic noise rather than replacing it: noise provides the perturbation to leave the saddle's neighbourhood, the metric provides the geometric correction that makes the surrounding bulk look convex.

9. The Gauss equation connection

The loss graph $\phi(\theta) = (\theta, L(\theta))$ is a submanifold of $\mathbb{R}^{N+1}$. As a submanifold it has two kinds of curvature: intrinsic curvature, detectable by inhabitants living on the surface, captured by the Riemann tensor of the induced metric; and extrinsic curvature, describing how the surface bends in the ambient space, captured by the second fundamental form.

For the loss embedding, the second fundamental form is proportional to $H$:

$$b_{ij} = \sqrt{\frac{\xi}{1 + \xi \|\ell\|^2_\gamma}}\,H_{ij}.$$

The Hessian of the loss is the shape operator of the loss surface. Via the Gauss equation, the intrinsic Riemann curvature encodes Hessian information through the second fundamental form. In two dimensions, the Gaussian curvature is

$$K = \frac{\xi \det(H)}{\det(\gamma)\,(1 + \xi \|\ell\|^2_\gamma)^2}.$$

Three cases: $K > 0$ if the Hessian has same-sign eigenvalues (bowl or dome), $K < 0$ if $H$ is indefinite (saddle), $K = 0$ if $H$ is singular. The qualitative structure of the Euclidean Hessian is faithfully reflected in the intrinsic geometry of the loss surface as a submanifold of the ambient space. This is why the framework feels canonical: the induced metric is not an arbitrary preconditioner choice but the natural geometric structure of the loss graph.

10. Empirical: the curvature mechanism is practically inactive (decoupled metrics)

The theory above says the diagonal metric can convert saddles into convex bulk when the curvature correction $C$ is engaged. The natural way to engage it explicitly is to split the single metric vector into two, decoupling the role that sets the parameter step from the role that drives the correction:

• $s^{\mathrm{precond}}_i$ controls the parameter update (escape-focused),
• $s^{\mathrm{curv}}_i$ controls the curvature correction $C$ (convexity-focused).

The hope was that both could be pursued at once: saddle escape and geodesic convexity. The probe (saddle and weighted-saddle losses; raw runs in results/saddle/, results/weighted_saddle/, and results/decoupled_metrics_probe.json) showed no benefit. The decoupled variant produced essentially identical trajectories to the escape-only variant, for three reasons.

1. The base gradient-magnitude drive ($\xi\,e^{s_i}\,g_i^2$) dominates the curvature correction ($\beta_c\,\mathrm{sign}(H_{ii})\,e^{s_i}\,g_i^2$) when $\beta_c \ll \xi$.
2. Both $s^{\mathrm{precond}}$ and $s^{\mathrm{curv}}$ saturate at the metric clip boundary ($\pm 4$) and converge to the same values.
3. On the $x^2 - y^2$ saddle the two metrics stayed within ${\approx}0.4$ of each other through step ${\sim}440$ and diverged only in a brief transient (peak ${\approx}4.9$ at step $480$) before both collapsed to the same clipped boundary by step $500$. The escape behaviour was identical to the escape-only variant (escape step $794$ for both, identical final loss $-6.069$).

The fundamental issue is timescale separation: the two drives operate at timescales ${\sim}1/(\mu\xi)$ and ${\sim}1/(\mu\beta_c)$. With $\beta_c/\xi \approx 0.1$ the correction is a 10% perturbation on the base drive, and the metric clip forces both to the same saturated state. For decoupled metrics to engage, the correction must be comparable in magnitude to the base drive ($\beta_c \sim \xi$), or the metric clip must be much larger, or the two metrics must run on genuinely different timescales — none of which were practical in the swept configurations.

This is the honest empirical status of the curvature mechanism. Theoretically the diagonal class flips eigenvalue signs in the bulk (Sections 4-7); at standard hyperparameter scales the correction term is swamped by the gradient-magnitude drive, and the mechanism is practically inactive as an independently controllable knob. It remains consistent with the implicit-sharpness behaviour documented in the learnable diagonal entry §9.2, which is the more durable practical signature of the same geometry.

Appendix A: Derivation of the Riemannian Hessian formula

The general Riemannian Hessian of a scalar function $L$ in coordinates is the classical expression

$$(\mathrm{Hess}_g L)_{ij} = \partial_i \partial_j L - \Gamma^k_{ij}(g)\,\partial_k L = H_{ij} - \Gamma^k_{ij}(g)\,\ell_k,$$

with $\ell_i = \partial_i L$ and $H_{ij} = \partial_i \partial_j L$. The first term is the Euclidean Hessian; the second is the correction that accounts for the curvature of the coordinate system through the Christoffel symbols $\Gamma^k_{ij}(g)$, which encode how the basis vectors twist as you move on the manifold. The Christoffel symbols of a metric $g$ are themselves built from a quantity we call the Christoffel bracket,

$$B_{ij,l}(g) := \partial_i g_{jl} + \partial_j g_{il} - \partial_l g_{ij}, \qquad \Gamma^k_{ij}(g) = \tfrac{1}{2}\,g^{kl}\,B_{ij,l}(g).$$

The bracket $B_{ij,l}$ is a particular symmetric combination of first derivatives of $g$. The full Christoffel symbol is obtained by contracting it with the inverse metric. The bracket is the "raw" object: it captures all the information about how the metric varies in space, and the contraction with $g^{kl}$ is the rescaling needed to make $\Gamma^k_{ij}$ transform correctly as a connection coefficient.

Applying the Riemannian Hessian formula to the induced metric $g_{ij} = \gamma_{ij}(\theta) + \xi\,\ell_i\,\ell_j$ requires computing $\Gamma^k_{ij}(g)$. The clean factorisation

$$(\mathrm{Hess}_g L)_{ij} = \frac{H_{ij} - C_{ij}}{1 + \xi\,\|\ell\|_\gamma^2}, \qquad C_{ij} = \Gamma^k_{ij}(\gamma)\,\ell_k$$

is not obvious from the formula for $\Gamma^k_{ij}(g)$. Both the $\gamma$-part and the $\xi$-part of the metric contribute to the bracket, and they combine through a Sherman-Morrison correction in the inverse metric. The derivation has two steps.

A.1 The Christoffel bracket splits cleanly

Split $g_{ij}$ as $\gamma_{ij} + \xi\,\ell_i\,\ell_j$ and use $\partial_i \ell_j = H_{ij}$,

$$\partial_i g_{jl} = \partial_i \gamma_{jl} + \xi\,(H_{ij}\,\ell_l + \ell_j\,H_{il}),$$ $$\partial_j g_{il} = \partial_j \gamma_{il} + \xi\,(H_{ij}\,\ell_l + \ell_i\,H_{jl}),$$ $$\partial_l g_{ij} = \partial_l \gamma_{ij} + \xi\,(H_{il}\,\ell_j + \ell_i\,H_{jl}).$$

Add the first two and subtract the third. The $\gamma$ terms reproduce the bracket of $\gamma$ alone,

$$B_{ij,l}(\gamma) := \partial_i \gamma_{jl} + \partial_j \gamma_{il} - \partial_l \gamma_{ij}.$$

The $\xi$ terms simplify by pairwise cancellation. The term $+\ell_j H_{il}$ from $\partial_i g_{jl}$ cancels $-H_{il}\ell_j$ from $-\partial_l g_{ij}$, and $+\ell_i H_{jl}$ from $\partial_j g_{il}$ cancels $-\ell_i H_{jl}$ from $-\partial_l g_{ij}$. What survives is $2\xi\,H_{ij}\,\ell_l$. Therefore

$$B_{ij,l}(g) = B_{ij,l}(\gamma) + 2\xi\,H_{ij}\,\ell_l. \tag{A.1}$$

The bracket of $g$ separates cleanly into a $\gamma$-part and an $H$-part with no cross terms. This is the first non-trivial fact.

The reason this works is the special structure of the $\xi$-piece of $g$. The term $\xi \ell_i \ell_j$ depends on $\theta$ only through $\ell$, and $\partial_i \ell_j = H_{ij}$. So all the derivative terms that arise involve $H_{ij}$, and the cyclic combination forces them to combine into the simple product $H_{ij} \ell_l$. A different non-loss-based embedding could give a more tangled bracket.

A.2 Contracting with the inverse metric and the gradient

The Christoffel symbol of $g$ is $\Gamma^k_{ij}(g) = \tfrac{1}{2}\,g^{kl}\,B_{ij,l}(g)$, and we need its contraction with $\ell_k$,

$$\Gamma^k_{ij}(g)\,\ell_k = \tfrac{1}{2}\,g^{kl}\,B_{ij,l}(g)\,\ell_k.$$

The inverse metric comes from Sherman-Morrison applied to $g = \gamma + \xi\,\ell\,\ell^\top$. This is applicable because $g$ is a rank-1 perturbation of $\gamma$:

$$g^{kl} = \gamma^{kl} - \frac{\xi\,(\gamma^{-1}\ell)^k\,(\gamma^{-1}\ell)^l}{1 + \xi\,\|\ell\|_\gamma^2}, \qquad \|\ell\|_\gamma^2 := \ell^\top \gamma^{-1} \ell. \tag{A.2}$$

We will need one intermediate identity. Contracting $g^{kl}$ twice with $\ell$,

$$\ell^\top g^{-1} \ell = \|\ell\|_\gamma^2 - \frac{\xi\,\|\ell\|_\gamma^4}{1 + \xi\,\|\ell\|_\gamma^2} = \frac{\|\ell\|_\gamma^2}{1 + \xi\,\|\ell\|_\gamma^2}. \tag{A.3}$$

Now contract (A.1) with $\tfrac{1}{2}\,g^{kl}\,\ell_k$ piece by piece.

From the $\xi$-part of (A.1), using (A.3),

$$\tfrac{1}{2}\,g^{kl}\,(2\xi\,H_{ij}\,\ell_l)\,\ell_k = \xi\,H_{ij}\,(g^{kl}\,\ell_k\,\ell_l) = \frac{\xi\,\|\ell\|_\gamma^2}{1 + \xi\,\|\ell\|_\gamma^2}\,H_{ij}. \tag{A.4}$$

From the $\gamma$-part of (A.1), plug (A.2) into $\tfrac{1}{2}\,g^{kl}\,B_{ij,l}(\gamma)\,\ell_k$,

$$\tfrac{1}{2}\,\gamma^{kl}\,B_{ij,l}(\gamma)\,\ell_k - \tfrac{1}{2}\cdot \frac{\xi}{1 + \xi\,\|\ell\|_\gamma^2}\,(\gamma^{-1}\ell)^k\,\ell_k\,(\gamma^{-1}\ell)^l\,B_{ij,l}(\gamma).$$

The first piece is exactly $\Gamma^k_{ij}(\gamma)\,\ell_k = C_{ij}$. In the second piece, $(\gamma^{-1}\ell)^k\,\ell_k = \|\ell\|_\gamma^2$, and $(\gamma^{-1}\ell)^l\,B_{ij,l}(\gamma) = \gamma^{kl}\,\ell_k\,B_{ij,l}(\gamma) = 2\,C_{ij}$ (twice the definition of $C$, because the factor of $1/2$ is absorbed). Combining,

$$C_{ij} - \frac{\xi\,\|\ell\|_\gamma^2}{1 + \xi\,\|\ell\|_\gamma^2}\,C_{ij} = \frac{C_{ij}}{1 + \xi\,\|\ell\|_\gamma^2}. \tag{A.5}$$

Adding (A.4) and (A.5),

$$\Gamma^k_{ij}(g)\,\ell_k = \frac{C_{ij}}{1 + \xi\,\|\ell\|_\gamma^2} + \frac{\xi\,\|\ell\|_\gamma^2}{1 + \xi\,\|\ell\|_\gamma^2}\,H_{ij}.$$

Substituting into the Riemannian Hessian formula,

$$(\mathrm{Hess}_g L)_{ij} = H_{ij} - \Gamma^k_{ij}(g)\,\ell_k = H_{ij}\,\bigl(1 - \tfrac{\xi\,\|\ell\|_\gamma^2}{1 + \xi\,\|\ell\|_\gamma^2}\bigr) - \frac{C_{ij}}{1 + \xi\,\|\ell\|_\gamma^2}.$$

The coefficient on $H_{ij}$ simplifies to $1/(1 + \xi\,\|\ell\|_\gamma^2)$, giving the boxed result

$$(\mathrm{Hess}_g L)_{ij} = \frac{H_{ij} - C_{ij}}{1 + \xi\,\|\ell\|_\gamma^2}.$$

In the fixed-$\gamma$ case, $B_{ij,l}(\gamma) = 0$ and so $C_{ij} = 0$, recovering $(\mathrm{Hess}_g L)_{ij} = H_{ij}/(1 + \xi\|\ell\|^2_\gamma)$, a positive scalar times the Euclidean Hessian. This is the Level 0 result.

The single factor $1/(1 + \xi\|\ell\|_\gamma^2)$ in the denominator comes from Sherman-Morrison; it is the same scalar that appears in the smooth-clipping update. The numerator splits cleanly because the $\xi$-bracket has the special form $H_{ij}\,\ell_l$, which is what allows the contraction with $g^{kl}\,\ell_k$ to factor through the scalar $\ell^\top g^{-1} \ell$. The $\gamma$-bracket contributes the new piece $C_{ij}$ that depends on $\nabla\gamma$ but not on $H$. That is the door to curvature correction.

Appendix B: Fixed off-diagonal extensions

Some variants of the induced metric add constant off-diagonal coupling vectors $a, b \in \mathbb{R}^N$ to the ambient metric, giving the pullback

$$g_{ij} = \gamma_{ij} + \xi\,\ell_i\,\ell_j + \xi\,(a_i\,\ell_j + \ell_i\,b_j),$$

with $\gamma, a, b$ all constant. This appendix carefully computes the bracket and shows that the sign-preservation conclusion holds only in the symmetric subset $a = b$. In the asymmetric case $a \ne b$, an additional non-scalar correction appears, and the Riemannian Hessian is no longer forced to inherit the eigenstructure of $H$.

For the calculation to make sense as a Riemannian construction, $g$ must be symmetric. Direct check: $g_{ij} - g_{ji} = \xi[(a_i - b_i)\ell_j - (a_j - b_j)\ell_i]$, which is nonzero whenever $a - b$ is not parallel to $\ell$. The bracket-based machinery below applies only when $a = b$ (or, more loosely, when the symmetric part of $g$ is used). The non-symmetric case is treated in the off-diagonal metric entry, where it is acknowledged as a non-Riemannian preconditioner rather than a metric.

Bracket derivation for $a = b$. Using $\partial_i \ell_j = H_{ij}$ and the constancy of $a, b$,

$$\partial_i (a_j\,\ell_l + \ell_j\,b_l) = a_j\,H_{il} + H_{ij}\,b_l,$$ $$\partial_j (a_i\,\ell_l + \ell_i\,b_l) = a_i\,H_{jl} + H_{ij}\,b_l,$$ $$\partial_l (a_i\,\ell_j + \ell_i\,b_j) = a_i\,H_{jl} + H_{il}\,b_j.$$

The cyclic sum $\partial_i g_{jl} + \partial_j g_{il} - \partial_l g_{ij}$ from the new term is

$$\xi\bigl[a_j H_{il} + 2 H_{ij} b_l - H_{il} b_j\bigr] = 2\xi\,H_{ij}\,b_l + \xi\,H_{il}\,(a_j - b_j).$$

With $a = b$, the second term vanishes and the total bracket reduces to

$$B_{ij,l}(g) = B_{ij,l}(\gamma) + 2\xi\,H_{ij}\,(\ell_l + b_l).$$

Define $w := \ell + b$. Then for constant $\gamma$ the bracket is $2\xi\,H_{ij}\,w_l$, the same structure as the rank-one loss embedding with $\ell$ replaced by $w$. Contracting with $\tfrac{1}{2}\,g^{kl}\,\ell_k$,

$$\Gamma^k_{ij}(g)\,\ell_k = \xi\,H_{ij}\,(g^{kl}\,\ell_k\,w_l) = \xi\,\beta(\theta)\,H_{ij},$$

giving $(\mathrm{Hess}_g L)_{ij} = H_{ij}\,(1 - \xi\,\beta(\theta))$. The Riemannian Hessian is $H$ multiplied by a single scalar, so eigenvalue signs are preserved (or globally flipped, exchanging minima and maxima but not turning an indefinite Hessian into a definite one). Fixed off-diagonal couplings with $a = b$ cannot enable sign flips.

The case $a \ne b$. When $a \ne b$, the ambient metric $h$ is not symmetric, so the pullback is not a Riemannian metric and the Riemannian Hessian construction does not apply. The construction in this case should be analysed as a non-symmetric preconditioner (see the off-diagonal entry Section 4 for the honest framing). The bracket calculation above still produces meaningful algebraic content, however: the extra term $\xi\,H_{il}\,(a_j - b_j)$ in the cyclic sum is not of the form "$H_{ij}$ times a vector contracted on $l$," so the analogue of the contraction $\Gamma^k_{ij}\,\ell_k$ in the symmetric case picks up a rank-one piece $\tfrac{1}{2}\,\xi\,(a_j - b_j)\,(H u)_i$ with $u^l = g^{kl}\,\ell_k$. This term has separate row and column structure and is not proportional to $H$. A direct generalisation of "Riemannian Hessian" to this non-Riemannian setting is not available; whether some symmetrised or "preconditioned-Hessian" eigenvalue analysis recovers a useful sign-flip statement here is open.

The factor of $H_{ij}$ that comes out of the cyclic cancellation is the algebraic obstruction in the symmetric case: as long as the bracket has the form "$H_{ij}$ times a vector contracted later," the Riemannian Hessian is forced to inherit the eigenstructure of $H$ unchanged. The asymmetric case breaks this factorisation but pays the price of no longer being a Riemannian construction.

Appendix C: Christoffel symbols of the diagonal metric

For the diagonal metric $\gamma_{mn}(\theta) = e^{s_m(\theta)}\,\delta_{mn}$, the inverse metric is $\gamma^{mn} = e^{-s_m}\delta^{mn}$, and the Christoffel bracket of $\gamma$ alone has components

$$B_{ij,l}(\gamma) = \partial_i \gamma_{jl} + \partial_j \gamma_{il} - \partial_l \gamma_{ij}.$$

Using $\gamma_{jl} = e^{s_j}\delta_{jl}$, so $\partial_i \gamma_{jl} = e^{s_j}\,\partial_i s_j\,\delta_{jl}$, this expands to

$$B_{ij,l}(\gamma) = e^{s_j}\,\partial_i s_j\,\delta_{jl} + e^{s_i}\,\partial_j s_i\,\delta_{il} - e^{s_i}\,\partial_l s_i\,\delta_{ij}.$$

Each term is supported only when specific indices coincide. Combining with $\gamma^{kl} = e^{-s_k}\delta^{kl}$, the Christoffel symbols $\Gamma^k_{ij}(\gamma) = \tfrac{1}{2}\gamma^{kl}B_{ij,l}(\gamma)$ are

$$\Gamma^k_{ij}(\gamma) = \tfrac{1}{2}\,e^{-s_k}\bigl(e^{s_j}\,\partial_i s_j\,\delta_{jk} + e^{s_i}\,\partial_j s_i\,\delta_{ik} - e^{s_i}\,\partial_k s_i\,\delta_{ij}\bigr).$$

Case-by-case:

(i) $i = j = k$. All three terms contribute and equal $e^{s_i}\,\partial_i s_i$, giving $\Gamma^i_{ii} = \tfrac{1}{2}\,\partial_i s_i$.

(ii) $i = j \ne k$. Only the third term survives; the $\delta_{ij}$ matches, the others do not. $\Gamma^k_{ii} = -\tfrac{1}{2}\,e^{s_i - s_k}\,\partial_k s_i$. This is where the exponential anisotropy $e^{s_i - s_k}$ first appears.

(iii) $i \ne j$, $k = j$. Only the first term survives. $\Gamma^j_{ij} = \tfrac{1}{2}\,\partial_i s_j$.

(iv) $i \ne j$, $k = i$. Only the second term survives. $\Gamma^i_{ji} = \tfrac{1}{2}\,\partial_j s_i$ (note the symmetry $\Gamma^i_{ij} = \Gamma^i_{ji}$ in the lower indices).

(v) $i \ne j$, $k \ne i$, $k \ne j$. All three terms vanish; the Kronecker deltas do not match. $\Gamma^k_{ij} = 0$.

C.1 Trace of the diagonal correction

Compute $C_{ii} = \sum_k \Gamma^k_{ii}(\gamma)\,\ell_k$ using the cases above,

$$C_{ii} = \tfrac{1}{2}\,\partial_i s_i\,\ell_i - \tfrac{1}{2}\sum_{k \ne i} e^{s_i - s_k}\,\partial_k s_i\,\ell_k.$$

Summing over $i$,

$$\mathrm{tr}(C) = \tfrac{1}{2}\sum_i \partial_i s_i\,\ell_i - \tfrac{1}{2}\sum_i \sum_{k \ne i} e^{s_i - s_k}\,\partial_k s_i\,\ell_k. \tag{C.1}$$

Sanity check: recover the scalar trace identity. Restrict to $s_i = s$ for all $i$ (the conformal case). Then $\partial_k s_i = \partial_k s$ and every $e^{s_i - s_k} = 1$, so

$$\mathrm{tr}(C) = \tfrac{1}{2}\,(\nabla s \cdot \ell) - \tfrac{1}{2}\,(N - 1)\,(\nabla s \cdot \ell) = \tfrac{2 - N}{2}\,(\nabla s \cdot \ell).$$

This matches Level 1.

Why no constraint without that restriction. In (C.1) without the conformal restriction, the exponential factors $e^{s_i - s_k}$ for $i \ne k$ are independent positive numbers (one for each ordered pair). Combined with the freely chosen $\partial_k s_i$ (the entries of the Jacobian $\sigma$), the right-hand side of (C.1) is a linear combination of $\ell_k$ with arbitrary positive-weighted coefficients. No algebraic identity forces it to vanish or to take a specific form. The trace is unconstrained.

C.2 The Gershgorin construction

With $\sigma_{ki} = \partial_k s_i$ chosen as $\sigma_{ii} = -2M/\ell_i$ and $\sigma_{ki} = 0$ for $k \ne i$, every Christoffel symbol that requires off-diagonal $\sigma$ vanishes. From case (ii), $\Gamma^k_{ii} = -\tfrac{1}{2}e^{s_i - s_k}\cdot 0 = 0$ for $k \ne i$. From cases (iii) and (iv), $\Gamma^j_{ij} = \tfrac{1}{2}\,\partial_i s_j = 0$ and $\Gamma^i_{ji} = \tfrac{1}{2}\,\partial_j s_i = 0$ for $i \ne j$, since $\partial_i s_j = \sigma_{ij} = 0$ in this construction.

Only the same-index $\Gamma^i_{ii} = \tfrac{1}{2}\sigma_{ii}$ survives. Therefore

$$C_{ii} = \Gamma^i_{ii}\,\ell_i = \tfrac{1}{2}\sigma_{ii}\,\ell_i = -M \text{ for every } i, \qquad C_{ij} = 0 \text{ for } i \ne j.$$

So $C = -M\,I$, and $H - C = H + M\,I$. Each diagonal entry of $H$ shifts up by $+M$ and off-diagonals are unchanged.

By Gershgorin's disc theorem, the eigenvalues of any matrix lie in discs centred at the diagonal entries with radii equal to the off-diagonal row sums. Shifting every disc centre up by $M$ while leaving the radii unchanged, for sufficiently large $M > \max_i (R_i - H_{ii})$, every disc lies strictly in the positive half-plane. Then every eigenvalue of $H - C$ is strictly positive and geodesic convexity is achieved.

The construction is not minimum-budget; it is a constructive proof of existence. The actual minimum-budget $\sigma$ is given by the SDP in Section 4.4.

Appendix D: Coordinate invariance

The Riemannian Hessian $(\mathrm{Hess}_g L)_{ij}$ transforms as a (0, 2)-tensor under coordinate changes. Its eigenvalues with respect to the metric $g$ (that is, the solutions of $\det(\mathrm{Hess}_g L - \lambda\,g) = 0$) are coordinate-invariant scalars. The sign-flip analysis above is therefore geometrically meaningful, not an artifact of the coordinate basis we used.

A practical implication: even though the diagonal-metric class $\gamma = \mathrm{diag}(e^{s_i})$ does single out the coordinate basis (it is diagonal in the parameter coordinates, not in some other basis), the questions it answers (can we achieve geodesic convexity? what is the budget threshold?) are coordinate-invariant. The choice of diagonal metric is a restriction on the class of $\gamma$ we are willing to use, not on the geometry of the question. A different metric class would give a different feasibility region; the diagonal class is chosen for its computational tractability ($O(N)$ storage and update cost) rather than for any geometric reason.

Appendix E: Attached materials

Notebooks (notebooks/):

• notebooks/derivations.nb: symbolic verification of the Christoffel bracket factorisation, the Sherman-Morrison contraction, the conformal trace identity, the case-by-case Christoffel symbols of the diagonal metric, and the trace formula (C.1) including the sanity-check reduction to Level 1.
• notebooks/sign-flip-sdp.nb: numerical SDP setup for $\max \lambda_{\min}(H - C)$ over $\|\sigma\|_F \le R$. Budget-vs-eigenvalue tables for the symmetric saddle, asymmetric saddle, monkey saddle, and Rosenbrock function. The $N = 5$ scalar-failure check. Sign-flip verification on diagonal Hessians with random off-diagonal perturbations for $N = 3, 5, 10, 50, 100, 1000$.
• notebooks/basin-enlargement.nb: computation and visualisation of $\mathcal{B}_{\mathrm{eucl}}$ versus $\mathcal{B}_g$ on the quartic well, including the four-basins-merge-into-one figure.
• notebooks/epsilon-excised-sign-flip.nb: symbolic derivation of Theorem 1ʹ (the budget-translated sign-flip criterion (6.3.1), the Taylor expansion at a Morse critical point, the cross-term lift and the uniform $\varepsilon$-bound) together with numerical verification of the predicted $\varepsilon(R)$ on the symmetric saddle (matches $4/R$ to scan resolution; sharp), the asymmetric saddle $L = x^2 - 3 y^2$, and the quartic well (failure region forms thin tubes around the 4 saddles and the maximum at the origin). The algebraic core is also formalised in Lean 4 in the project repository (zero sorry, six documented axioms).

Scripts (scripts/):

• scripts/make_figures.py: reproduces every figure shown in this entry (PDF and PNG written to figures/).