The Master Minimax Framework and the Tautology Test

The induced-metric framework gives a clean family of optimizers, parameterised by the choice of embedding. Each choice produces a preconditioner; each preconditioner has a worst-case convergence rate on some class of loss landscapes. A natural reframing of the optimisation question is to fix a class of landscapes and ask which preconditioner is provably best, in a precise minimax sense, on that class.

This entry sets up that reframing as a meta-theorem template (the master minimax) and identifies a structural pitfall that nearly any "optimality" theorem in this style falls into (the tautology test). Two existing optimality results from the project (the diagonal minimax in the derived optimizer and the Kronecker minimax in the pullback unification entry) are run through the test as worked examples, and both turn out to be tautological. This does not invalidate them, but it does change what they should be advertised as: they are algorithmic derivations of known optimizers from the embedding geometry, not novel optimality claims.

1. Why this reframing

After running a year of experiments, the project's empirical evidence does not support a clean "this is a better optimizer than Adam" claim (see the empirical sections of the individual method entries, e.g. the learnable diagonal §9 and the base induced metric §9-11). But the embedding framework still has structural content that does not depend on whether a particular hyperparameter sweep wins. The right question to ask of a theoretical framework is not

Can we find a better optimizer?

but rather

Given a problem class $\mathbb{C}$, a preconditioner class $\mathcal{F}$, and a rate functional $\rho$, what is the optimizer that is provably optimal under those assumptions?

This is a meta-question about the framework, not a benchmark question. Every concrete optimality theorem in the program is an instance of it. The point of writing down the template is to make the assumptions explicit, so that the reader can immediately see what an "optimality" claim is and is not asserting.

2. The master minimax statement

Fix three objects.

Preconditioner class $\mathcal{F}$. A set of embeddings (or, equivalently, base metrics $\gamma$, ambient metrics $h$, and embedding maps $f$) that the optimizer is permitted to use. Each $\varphi \in \mathcal{F}$ determines a pullback metric $G(\varphi, \theta)$ and an update rule

$$\delta\theta = -\eta\, G(\varphi, \theta)^{-1}\, \nabla L(\theta).$$

Problem class $\mathbb{C}$. A family of loss landscapes, or, for quadratic local analysis, a family of symmetric positive definite Hessians.

Rate functional $\rho(\varphi, L)$. A scalar measure of optimizer performance on a specific loss. Typical choices are the worst-case condition number $\kappa(G^{-1/2} H G^{-1/2})$, the asymptotic contraction rate, the iteration complexity to reach $\varepsilon$-accuracy, or a generalisation gap.

The optimal pullback embedding for the triple $(\mathcal{F}, \mathbb{C}, \rho)$ is

$$\varphi^*(\mathcal{F}, \mathbb{C}, \rho) = \underset{\varphi \in \mathcal{F}}{\arg\min}\,\sup_{L \in \mathbb{C}}\,\rho(\varphi, L).$$

This is the single equation that the framework revolves around. Every optimality theorem in the program is an instance: pin down $(\mathcal{F}, \mathbb{C}, \rho)$, solve for $\varphi^*$, and compare the resulting optimizer to known ones.

3. The tautology test

The trap is the following. If the preconditioner class is rich enough to invert every element of the problem class, then the minimax trivially returns the per-instance inverse and there is no real optimisation happening. We have asked a class that contains every answer to produce the answer, and unsurprisingly it does.

Definition. An instance $(\mathcal{F}, \mathbb{C}, \rho)$ is tautological if for every $H \in \mathbb{C}$ there is a $\varphi \in \mathcal{F}$ with $G(\varphi) \propto H$ (equivalently, $G(\varphi)^{-1} \propto H^{-1}$), where the proportionality absorbs the learning-rate gauge, and $\rho$ is monotone in $\kappa(G^{-1} H)$.

When the test flags tautology (the condition above holds), the minimax-optimal $\varphi^*$ is by construction the unique element (up to gauge) that satisfies $G(\varphi^*) \propto H$ on each $H$, giving preconditioner $G(\varphi^*)^{-1} \propto H^{-1}$. The theorem then states "the optimal preconditioner is the exact inverse of the Hessian," which is true but uninformative.

4. Two worked examples, both tautological

The project's two existing optimality theorems both fail the tautology check. They are still useful as algorithmic derivations, but the framing needs to be honest.

4.1 Diagonal minimax (s-target theorem)

• $\mathcal{F} = \{G = \mathrm{diag}(e^s) : s \in \mathbb{R}^N,\,\sum_i s_i = 0\}$, where $G$ is the metric and $G^{-1}$ is the preconditioner applied to the gradient (matching the convention of Section 2). The mean-centering constraint $\sum s_i = 0$ is a gauge fix that removes the overall scale degeneracy with the learning rate.
• $\mathbb{C} = \{H = \mathrm{diag}(h) : h \in \mathbb{R}_{>0}^N,\, h_{\max}/h_{\min} \le \kappa_0\}$.
• $\rho = \kappa(G^{-1} H) = \max_i (e^{-s_i} h_i) / \min_i (e^{-s_i} h_i)$.

The tautology check flags this instance: for any diagonal $H$, the choice $s_i = \log h_i - \overline{\log h}$ (the mean-centered form required by the gauge constraint, with the residual scalar absorbed into the learning rate) makes $G \propto H$ and the preconditioner $G^{-1} \propto H^{-1}$, exactly the Newton step. The minimax reduces to "equalise $e^{-s_i} h_i$ across $i$," giving

$$s_i^* = \log h_i - \overline{\log h}, \qquad \kappa(G^{*-1} H) = 1.$$

Sign convention note. The derived optimizer uses the opposite convention: it parameterises the preconditioner directly as $M = \mathrm{diag}(e^{s_i})$ (no inverse), so its Newton formula reads $s_i^* = -\log h_i + \overline{\log h}$. The two formulas describe the same theorem; they only differ by which object $s$ parameterises (metric versus preconditioner). When cross-referencing, flip the sign of $s$.

This is the result that drives the derived optimizer. It is genuinely useful: it identifies the algorithmic form of the Newton diagonal as the unique minimax-optimal log-diagonal scaling, and forces the mean-centering convention. But as a minimax theorem it says "if the preconditioner class can invert the Hessian, the optimal thing to do is invert it." That is the definition of the class.

4.2 Kronecker minimax (KFAC theorem)

• $\mathcal{F} = \{G = P \otimes Q : P \in \mathrm{SPD}(m),\, Q \in \mathrm{SPD}(n),\, \det(P)^n \det(Q)^m = 1\}$, with $G$ the metric and $G^{-1}$ the preconditioner. The volume constraint is a gauge fix on the scale degeneracy.
• $\mathbb{C} = \{H = A \otimes B : A \in \mathrm{SPD}(m),\, B \in \mathrm{SPD}(n)\}$.
• $\rho = \kappa(G^{-1} H)$.

The tautology check returns $\mathcal{F}^{-1} \supseteq \mathbb{C}$: for any Kronecker $H = A \otimes B$, the choice $P \propto A$ and $Q \propto B$ gives $G \propto H$, hence preconditioner $G^{-1} \propto H^{-1}$; exactly the Newton step on the Kronecker block. The minimax reduces to "equalise the $mn$ pairwise products $\lambda_i(P^{-1} A) \cdot \mu_j(Q^{-1} B)$," and the rank-one factorisation argument (Section 7 of the pullback unification entry) forces $P^* \propto A$ and $Q^* \propto B$.

Again the theorem is useful: it gives an algorithmic derivation of KFAC from minimax over the Kronecker class, independent of the Fisher-approximation derivation in Martens and Grosse. But it is also tautological: the Kronecker class can invert any Kronecker Hessian, and minimax forces it to do so.

5. Why tautological theorems are still publishable

Tautology is a property of the class choice, not of the theorem's value. Both worked examples are real contributions in the following sense: they identify the unique algorithmic form that satisfies a clean optimality criterion on a clean problem class. Without the theorems we would not know that the natural form of the Newton diagonal is

$$s_i = \log H_{ii} - \overline{\log H} \quad \text{(metric convention)}, \qquad \text{equivalently} \quad s_i = -\log H_{ii} + \overline{\log H} \quad \text{(preconditioner convention)}$$

(the mean-centering term is not arbitrary; it is forced by the gauge constraint), or that the natural form of the Kronecker-factored metric approximation (whose inverse is the applied preconditioner) is

$$M = \det(A)^{-1/m} A \otimes \det(B)^{-1/n} B$$

(the normalising constants are forced by the volume constraint, which is itself just a gauge fix; the individual factors $P, Q$ are not separately determined since $(P, Q) \to (cP, c^{-1}Q)$ leaves $P \otimes Q$ invariant).

A tautological theorem can still surprise. What it cannot do is demonstrate that the framework produces optimizers that beat known methods on the chosen problem class, because by construction the framework's optimum is the per-instance inverse, which is also the per-instance optimum of any reasonable rate functional. To demonstrate novelty in that sense, the problem class needs to be richer than the preconditioner class.

6. Escaping tautology

There are four structural ways to make $\mathcal{F}^{-1} \not\supseteq \mathbb{C}$.

Rank restriction. $\mathcal{F}$ has lower expressive rank than $\mathbb{C}$. The cleanest example is the spike-bulk minimax of Section 8: diagonal preconditioner ($\mathcal{F}$) on a Hessian with a single rank-one off-diagonal spike ($\mathbb{C}$).

Structural restriction. $\mathcal{F}$ is constrained to a specific symmetry or sparsity pattern that $\mathbb{C}$ does not share. For example, restrict $\mathcal{F}$ to block-diagonal but let $\mathbb{C}$ contain dense Hessians.

Information restriction. $\varphi \in \mathcal{F}$ is chosen using only observables of $L$ (gradients, Hutchinson Hessian-vector-product samples), not $H$ itself. This is the usual setting for an online optimizer, and is what makes a stochastic version of even a tautological class non-tautological in practice.

Cost restriction. $\mathcal{F}$ is parameterised with bounded compute or memory: representing $H^{-1}$ exactly exceeds the budget. This is the situation for any finite-time optimizer that maintains less state than a full Hessian.

The common theme: the adversary (the worst-case $H \in \mathbb{C}$) can produce curvature structure that the preconditioner cannot match, so the optimizer must approximate rather than invert. The interesting theorems are about the structure of that approximation.

7. Scope: what counts as a pullback optimizer

The framework covers update rules of the form

$$\delta\theta_t = -\eta\, G(\varphi, \theta_t)^{-1}\, \nabla L(\theta_t)$$

where $G$ is determined by the embedding $\varphi$ and depends only on the current $\theta_t$ (plus possibly auxiliary metric-learning state $s_t$ that evolves deterministically from the history of $\{\theta, \nabla L\}$). The update at step $t$ is memoryless given $G_t$. No look-ahead, no velocity state, no first-moment accumulator.

7.1 In-scope

Under the convention $\delta\theta = -\eta\, G^{-1}\nabla L$ of Section 2, with $G$ the metric and $G^{-1}$ the applied preconditioner:

• SGD ($G = I$, any $\varphi$).
• Newton ($G = H$, so $G^{-1} = H^{-1}$ is applied).
• Natural gradient ($G = F$ for Fisher $F$, so $G^{-1} = F^{-1}$ is applied).
• Gauss-Newton ($\varphi = (\theta, \sqrt{\xi}\,J(\theta)^\top \cdot \text{output})$, giving $G = I + \xi J^\top J$ which is proportional to $H_{\mathrm{GN}}$ in the $\xi \to \infty$ limit).
• KFAC (Kronecker factorisation of the Gauss-Newton pullback, with $G = P \otimes Q$ and applied preconditioner $G^{-1} = P^{-1} \otimes Q^{-1}$).
• Learnable scalar or diagonal metric (with $s$ evolving via gradient ascent or EMA on observables).
• Hutchinson-based curvature estimation (HVP-driven state).

7.2 Out of scope

These optimizers are excluded by construction, not by accident: they are not pullback metrics on $\theta$ alone, regardless of which embedding is chosen.

• Nesterov acceleration: depends on a look-ahead gradient at $\theta + \beta v$, not captured by a metric acting on $\nabla L(\theta)$.
• Heavy-ball momentum: carries velocity $v_t = \beta v_{t-1} - \nabla L$, trajectory-level state on $\theta$ that is not expressible as $G^{-1} \nabla L$ for any $G$.
• Adam's first-moment EMA / momentum numerator: the bias-corrected $\hat m_t$ acts as a velocity-like state on the gradient trajectory and cannot be written as $G(\theta_t)^{-1} \nabla L(\theta_t)$ for any metric $G$. (The second-moment EMA on its own, as in RMSProp or Adam with $\beta_1 = 0$, is a valid pullback metric: it parameterises a diagonal metric whose entries evolve via slow-timescale EMA of $g_i^2$, which §7.3 explicitly allows. What violates the memoryless-gradient form is the first-moment accumulator on the gradient itself.)
• Trust-region and line-search methods: choose step size by evaluating $L$ at candidate points, not a one-step metric update.
• Signed updates (Lion-style): $\mathrm{sign}(\cdot)$ is not induced by any inner product.

This is an honest restriction. The framework is set up to describe one family of optimizers and does not claim to subsume momentum-style methods. The method entries' empirical sections compare against Adam, SGD, Muon, AdamW, and several momentum baselines for context, but no optimality theorem in this program claims to dominate them.

7.3 "Memoryless given $G$" in detail

The framework allows the metric $G$ itself to carry slow-timescale state, such as a learnable diagonal $s$ evolving via an EMA of curvature estimates. What is excluded is fast-timescale state on $\theta$'s trajectory: velocity, momentum, look-ahead. The restriction is honest and carves out the subset of optimizers the framework is actually equipped to describe.

8. The first non-tautological instance: spike-bulk minimax

The first interesting application of the master template is a minimax problem whose tautology check fails. The setup.

• $\mathcal{F} = \{G = \mathrm{diag}(e^s) : s \in \mathbb{R}^N,\,\sum_i s_i = 0\}$ (diagonal, same as in Section 4.1).
• $\mathbb{C} = \{H = D + v v^\top : D \in \mathrm{diag}_{>0},\,v \in \mathbb{R}^N,\,\lvert\mathrm{supp}(v)\rvert \ge 2,\,\lVert v\rVert \le R,\,h_{\max}(D)/h_{\min}(D) \le \kappa_0\}$.
• $\rho = \kappa(G^{-1/2} H G^{-1/2})$.

The tautology check now fails. By Sherman-Morrison,

$$H^{-1} = D^{-1} - \frac{D^{-1} v v^\top D^{-1}}{1 + v^\top D^{-1} v}.$$

The second term has at least one nonzero off-diagonal entry (whichever $(i, j)$ has $v_i v_j \ne 0$). No $G = \mathrm{diag}(e^s) \in \mathcal{F}$ has off-diagonal entries, so $H^{-1} \notin \mathcal{F}$. The instance is non-tautological.

The next theorem in the program (in progress) is to solve this minimax explicitly: what is the optimal diagonal scaling $s^*(D, v)$ given a rank-one off-diagonal spike, and how badly does it underperform the (unavailable) exact inverse? The answer should quantify how much off-diagonal structure the diagonal class can absorb, and where it cannot.

9. The rate functional in Phase 1: worst-case condition number

For the first non-tautological instance, the rate functional is

$$\rho(\varphi, H) = \kappa\bigl(G(\varphi)^{-1/2}\, H\, G(\varphi)^{-1/2}\bigr).$$

This functional has four useful properties. It is scale-invariant ($\kappa$ is invariant under $G \to c G$). It is monotone in the standard contraction-rate functional for gradient descent on a quadratic (so a smaller $\kappa$ is a tighter convergence-rate bound). It is tractable symbolically for rank-one and small-rank perturbations via the secular equation. And it is the functional used in both worked examples in Section 4, so the master notebook can reproduce those as sanity checks under the same $\rho$.

Later instances may use different functionals: iteration complexity, stochastic-convergence rate, generalisation-gap upper bound, or a hybrid. The Phase 1 instance uses $\kappa$ to minimise the conceptual distance from known results.

10. Workflow for applying the template

Every theorem in the program follows the same recipe.

Step 1. State $(\mathcal{F}, \mathbb{C}, \rho)$ precisely. $\mathbb{C}$ must be a family of Hessians or losses with a structural parameter (dimension, rank, condition-number bound). $\mathcal{F}$ must be a subset of embeddings described by a parameter vector. $\rho$ must be a scalar function of $(\varphi, L)$.

Step 2. Run the tautology check. Verify whether $\mathcal{F}^{-1} \supseteq \mathbb{C}$. If yes, the theorem is a re-derivation of a known optimizer, not a new one. Still publishable as a unification, but the framing must say so.

Step 3. Compute the rate symbolically. For quadratic $L$ this reduces to a spectral calculation on $G^{-1/2} H G^{-1/2}$. Sherman-Morrison and Woodbury make low-rank cases tractable.

Step 4. Solve the minimax. For tractable instances this yields a closed-form $\varphi^*$ as a function of the problem-class parameters. Otherwise characterise $\varphi^*$ via KKT conditions.

Step 5. Verify numerically. Plug specific rational parameter values into a small numerical script and confirm that the symbolic $\varphi^*$ is the true minimum over a large random sample of competitors.

Step 6. Scenic-route writeup. Narrative explanation in the project's house style, with derivations shown rather than asserted.

11. What the framework does and does not claim

What the framework does claim, after this entry.

• A subset of well-known optimizers (SGD, Newton, GN, Fisher, natural gradient, KFAC, and learnable scalar or diagonal variants) are pullback metrics under explicit embedding choices, and can be derived inside a single template.
• For tautological problem-preconditioner classes, the minimax-optimal pullback recovers the per-instance inverse, providing an algorithmic form (with gauge fixings) for the optimizer.
• For non-tautological classes, the minimax-optimal pullback is the structurally best approximation, and the gap between approximation and inverse is the substantive content of the theorem.

What the framework does not claim.

• That every reasonable optimizer is a pullback metric. Momentum, Nesterov, Adam, and trust-region methods are excluded by construction (Section 7).
• That tautological optimality theorems demonstrate novelty in the "this is a better optimizer than X" sense. They demonstrate the algorithmic form, not the performance.
• That the empirical ranking of induced-metric variants against Adam/Muon/AdamW reflects the framework's theoretical content. The two are decoupled. See the method entries' empirical sections for what the experiments actually show.

12. Position within the program

The framework is the bridge between two halves of the project. On the theory side, it organises every optimality theorem (diagonal, Kronecker, spike-bulk, and future extensions) under one template, with a built-in honesty check. On the empirics side, it makes the structural assumptions explicit so the reader can see exactly what assumption fails when a derived optimizer underperforms in practice (typically the structural assumption on $\mathbb{C}$, as in the derived optimizer's failure analysis).

The next theorem in the program is the spike-bulk minimax of Section 8, whose tautology check fails. After that, the natural extensions are to information-restricted classes (preconditioner uses only HVP samples, not $H$) and to cost-restricted classes (preconditioner has a fixed memory budget).

Appendix A. Tautology check for the Hutchinson-driven estimator

A subtler version of the tautology question arises when the preconditioner is chosen using observables rather than $H$ itself. Even if the deterministic version of the class can invert every $H$, the stochastic version (where $\hat H_{ii}$ is estimated from $K$ Hutchinson HVP samples) is non-tautological: the estimator has finite variance, so the achieved preconditioner is a random approximation of the deterministic minimax-optimal one, and the rate functional must account for the variance.

The robustness theorem in the derived optimizer (Appendix B) addresses exactly this question for the diagonal case. The version for the Kronecker case (relative error on each factor) is straightforward but has not been written up.

Appendix B. The volume constraint as a gauge fix

In both the diagonal and Kronecker minimax theorems, the preconditioner class is restricted by a determinant constraint ($\sum s_i = 0$ in the diagonal case, $\det(P)^n \det(Q)^m = 1$ in the Kronecker case). These are not substantive assumptions; they fix the gauge degeneracy that comes from the symmetry

$$(G, \eta) \to (c G,\, c \eta),$$

which leaves the update $\delta\theta = -\eta\, G^{-1} \nabla L$ invariant. Without a gauge fix, the minimax problem has a one-parameter family of equivalent solutions, and the theorem statement would have to quotient by this. The constraint just picks a canonical representative.

Appendix C. Numerical infrastructure

The companion materials.

• repo/notebooks/hp-minimax-master.nb: Mathematica module implementing Embedding[], Pullback[], Rate[], and Minimax[]. Reproduces the diagonal and Kronecker theorems as worked examples and computes the spike-bulk minimax symbolically for small $N$.

• scripts/hp_tautology_check.py: Numerical utility that takes samplers for $\mathcal{F}$ and $\mathbb{C}$, computes the Frobenius distance from each $H^{-1}$ to its projection onto $\mathcal{F}^{-1}$, and reports the empirical tautology check (the test fails when this distance is consistently positive).

Appendix D. Attached materials

Scripts (scripts/):

• scripts/hp_tautology_check.py: numerical verification of tautology checks for arbitrary $(\mathcal{F}, \mathbb{C})$ pairs.
• scripts/make_figures.py: reproduces every figure shown in this entry (PDF and PNG written to figures/).

Notebooks (notebooks/):

• notebooks/hp-minimax-master.nb: symbolic implementation of the master template (Frobenius projection, tautology check, gauge-fixed Newton-target solver).
• notebooks/worked-examples.nb: the diagonal, Kronecker, and spike-plus-bulk minimax instances run through the master template.

References

• Amari, S. (1998). Natural gradient works efficiently in learning. Neural Computation 10, 251-276.
• Harvey, T. R. (2025). The optimiser hidden in plain sight: training with the loss landscape's induced metric. arXiv:2509.03594.
• Martens, J., Grosse, R. (2015). Optimizing neural networks with Kronecker-factored approximate curvature. ICML.