Embedding parameter space via the network output (rather than the scalar loss) and pulling back the ambient metric reproduces the regularized Gauss-Newton method, Fisher information, the natural gradient, and — under one further Kronecker approximation — KFAC. The induced-metric framework is not introducing a new optimizer here; it provides a single geometric origin for several optimizers that are usually derived separately.
Typed links between this entry and other entries in the knowledge graph.
Runs the Kronecker minimax (KFAC) through the tautology test; finds it tautological.
KFAC is the Kronecker structural upgrade of the derived optimizer's diagonal preconditioner.
Output-embedding pullback generalises the loss-embedding induced metric; recovers Gauss-Newton/Fisher/natural-gradient/KFAC.