Information Geometry

What does it mean for two probability distributions to be "close"? This seemingly simple question leads to a profound mathematical framework: information geometry: the study of families of probability distributions as curved geometric spaces. By equipping statistical models with a natural Riemannian metric (the Fisher information metric), information geometry reveals deep structural connections between statistics, thermodynamics, quantum mechanics, and potentially gravity itself.

Information geometry is a bridge field. It connects pure mathematics (differential geometry, Riemannian manifolds) with theoretical physics (thermodynamics, quantum theory) and modern applications (machine learning, neural networks). Its central objects, the Fisher metric, dual connections, and divergence functions, appear independently in each of these fields, suggesting a universal geometric structure underlying inference and physics.

The Fisher Information Metric

The Fisher information metric is the unique (up to scaling) Riemannian metric on a statistical manifold that is invariant under sufficient statistics. Given a family of probability distributions $p(x|\theta)$, it is defined as:

This metric has a clear operational meaning: it measures the sensitivity of the probability distribution to changes in the parameters. If a small change in $\theta$ produces a large change in the distribution, the Fisher metric is large in that direction.

Why This Metric Is Special

Chentsov's theorem (1972) establishes that the Fisher metric is the unique Riemannian metric (up to a constant factor) that is invariant under Markov embeddings (sufficient statistics). This uniqueness theorem is analogous to the uniqueness of the Haar measure on groups, it tells us that the Fisher metric is not just a convenient choice but the only natural choice for measuring distances between probability distributions.

Example: The Normal Distribution

For the Gaussian family $p(x|\mu,\sigma) = \frac{1}{\sqrt{2\pi}\sigma} \exp\left(-\frac{(x-\mu)^2}{2\sigma^2}\right)$ with parameters $\theta = (\mu, \sigma)$, the Fisher metric is:

The resulting geometry is the Poincare upper half-plane: a space of constant negative curvature $K = -1/2$. This means the space of Gaussian distributions is hyperbolic: distributions with small variance (high precision) are far apart in the Fisher metric, even if their means are close. This reflects the intuition that distinguishing two sharp distributions is easy, while distinguishing two broad ones is hard.

The Cramer-Rao Bound

The Fisher information metric has a direct operational significance through the Cramer-Rao bound, one of the most fundamental results in statistics:

This bound connects geometry to measurement. The larger the Fisher information (the more curved the manifold), the more precisely we can estimate parameters. This perspective has been transformative in quantum metrology, where the quantum Fisher information sets fundamental bounds on measurement precision.

Statistical Manifolds and Dual Connections

Amari's key insight (1985) was that statistical manifolds naturally carry two affine connections, not just one:

where $\ell = \log p(x|\theta)$. These are called the exponential connection $\nabla^{(e)}$ and the mixture connection $\nabla^{(m)}$. They are dual with respect to the Fisher metric:

Dually Flat Structures

Exponential families (Gaussian, Poisson, Bernoulli, etc.) are flat under the exponential connection, while mixture families are flat under the mixture connection. This dually flat structure leads to remarkable results:

• Pythagorean theorems: Projections onto exponential or mixture submanifolds satisfy a Pythagorean relation with respect to the KL-divergence.
• Bregman divergences: The KL-divergence $D_{\text{KL}}(p \| q)$ is a Bregman divergence associated with the dually flat structure.
• Canonical coordinates: The natural and expectation parameters provide two dual coordinate systems in which the respective connections are flat.

Quantum Information Geometry

When we pass from classical to quantum probability, the geometric picture becomes richer and more subtle. Quantum states (density matrices $\rho$) form a manifold, and the question "how distinguishable are two quantum states?" again leads to a Riemannian metric, but now there are multiple natural choices.

The Fubini-Study Metric

For pure quantum states $|\psi\rangle$ in a Hilbert space of dimension $d$, the natural metric on the projective space $\mathbb{C}P^{d-1}$ is the Fubini-Study metric:

This metric measures the "angle" between quantum states and has a direct operational meaning: it determines the speed of quantum evolution (the Anandan-Aharonov relation) and sets bounds on quantum computation speed.

Quantum Fisher Information

For mixed states (density matrices $\rho(\theta)$), the quantum analog of the Fisher information is not unique. The most common choices are:

• Symmetric Logarithmic Derivative (SLD) metric: Defined through the operator $L$ satisfying $\partial_i \rho = \frac{1}{2}(\rho L_i + L_i \rho)$: $$g^{\text{SLD}}_{ij} = \frac{1}{2}\text{Tr}[\rho(L_i L_j + L_j L_i)]$$
• Bures metric: The infinitesimal version of the Bures distance (fidelity-based distance) between density matrices.
• Wigner-Yanase information: $I(\rho, A) = -\frac{1}{2}\text{Tr}[\sqrt{\rho}, A]^2$, measuring the "information content" of an observable $A$ in state $\rho$.

The quantum Cramer-Rao bound using the SLD metric gives the ultimate precision limit for parameter estimation in quantum mechanics, the Heisenberg limit:

where $F_Q$ is the quantum Fisher information, $N$ is the number of measurements, and $H$ is the Hamiltonian generating the parameter-dependent evolution. This is central to quantum metrology and quantum sensing.

Thermodynamic Geometry

Ruppeiner (1979) and Weinhold (1975) independently discovered that thermodynamic state spaces carry natural Riemannian metrics. Ruppeiner's metric is defined as:

where $S$ is the entropy and $X^i$ are the extensive variables (energy, volume, particle number). This is precisely the Fisher information metric for the Boltzmann distribution, the connection between information geometry and thermodynamics is exact.

Black Hole Thermodynamic Geometry

Since black holes are thermodynamic objects (with temperature, entropy, and free energy), they have a Ruppeiner geometry. For the Kerr-Newman black hole (charged, rotating), the Ruppeiner curvature reveals:

• The Schwarzschild black hole has zero Ruppeiner curvature: its thermodynamic manifold is flat, corresponding to the absence of microscopic "interactions" (in a sense that remains to be fully understood).
• Charged (Reissner-Nordstrom) and rotating (Kerr) black holes have nonzero curvature, with divergences at extremality, hinting at phase transition-like behavior.
• The curvature changes sign for certain parameter ranges, suggesting transitions between attractive and repulsive effective interactions among the microscopic degrees of freedom.

This raises a tantalizing question: can the Ruppeiner geometry of black holes teach us about the microscopic structure of quantum gravity?

Natural Gradient in Machine Learning

Amari (1998) introduced the natural gradient: gradient descent that respects the information geometry of the parameter space. Standard gradient descent treats parameter space as flat (Euclidean), but neural network parameters define probability distributions, and the natural metric on this space is the Fisher metric.

where $g^{-1}$ is the inverse Fisher information matrix. The natural gradient pre-multiplies the ordinary gradient by the inverse Fisher metric, accounting for the curvature of the statistical manifold. This is more efficient because it moves in the direction of steepest descent in the space of distributions, not in the space of parameters.

Modern practical methods (K-FAC, Fisher-Rao regularization, natural policy gradient in reinforcement learning) are all descendants of this information-geometric insight.

Toward Information-Geometric Gravity

The deepest and most speculative connection is between information geometry and gravity. Several threads suggest this connection:

• Emergent gravity from entanglement: If spacetime emerges from entanglement (as suggested by the Ryu-Takayanagi formula and ER=EPR), then the geometry of spacetime might be the Fisher geometry of some underlying statistical or quantum system.
• Fisher metric on the space of geometries: The space of all Riemannian metrics on a manifold (the DeWitt superspace) itself carries a metric (the DeWitt metric). Is there a connection to the Fisher metric on the space of quantum states?
• Thermodynamic geometry of black holes: The Ruppeiner metric on black hole thermodynamic state space is literally the Fisher metric. As black holes become better understood through holography, this connection may deepen.
• AdS/CFT and Fisher information: The bulk metric in AdS/CFT can be related to the Fisher information of the boundary CFT, providing a concrete realization of the idea that spacetime geometry encodes statistical distinguishability.

Key Insights

• Families of probability distributions form Riemannian manifolds with the Fisher information metric, the unique natural metric invariant under sufficient statistics.
• The Cramer-Rao bound connects geometry to measurement: the Fisher metric sets the ultimate precision limit for parameter estimation.
• Statistical manifolds carry dual connections (exponential and mixture) whose dually flat structure underlies the Legendre transform of thermodynamics.
• Quantum information geometry extends these ideas to quantum states, with the quantum Fisher information setting the Heisenberg limit for quantum metrology.
• Ruppeiner's thermodynamic geometry shows that curvature encodes microscopic interactions, flat for ideal gases, divergent at phase transitions.
• Black hole thermodynamic geometry provides a window into the microscopic structure of quantum gravity.
• The natural gradient (Amari) applies information geometry to machine learning, yielding more efficient optimization.
• The deepest open question is whether spacetime geometry itself is an information-geometric object, the Fisher metric of some fundamental quantum system.