Information & Entropy

In 1948, Claude Shannon founded information theory with a mathematical definition of information that turned out to be identical in form to Boltzmann's entropy. This was not a coincidence. Over the following decades, physicists and information theorists have established that information is physical -- it is stored in physical systems, processed by physical operations, and its manipulation is subject to thermodynamic constraints. This connection between information and thermodynamics may be the deepest unifying theme in all of physics.

Shannon Entropy

Shannon asked: given a probability distribution over possible messages, what is the minimum number of bits needed, on average, to encode a message? His answer was a quantity he called entropy:

The properties of Shannon entropy are striking:

• $H \geq 0$, with equality only for a certain outcome
• $H$ is maximized by the uniform distribution: $H_{\max} = \log_2 N$ for $N$ outcomes
• $H$ is additive for independent systems
• $H$ is continuous in the probabilities

Shannon proved that these properties uniquely determine the functional form $-\sum p_i \log p_i$. Any reasonable measure of uncertainty must take this form.

Connection to Boltzmann Entropy

Compare Shannon's formula with the Gibbs entropy of statistical mechanics:

$$S_{\text{Gibbs}} = -k_B \sum_i p_i \ln p_i$$

They are identical up to a choice of logarithm base and a multiplicative constant ($k_B$ and the factor $\ln 2$). For the microcanonical ensemble, where all $\Omega$ microstates are equally probable ($p_i = 1/\Omega$), this reduces to Boltzmann's formula:

$$S = k_B \ln \Omega$$

This is the same mathematics applied in two different contexts. The Gibbs entropy measures our ignorance about which microstate the system occupies. The Shannon entropy measures our ignorance about which message will be sent. The identification is not metaphorical -- it is exact.

Maxwell's Demon

In 1867, James Clerk Maxwell imagined a tiny intelligent being -- a "demon" -- who could observe individual molecules in a box of gas and sort them: fast molecules to one side, slow molecules to the other. This would create a temperature difference without doing any work, apparently violating the second law of thermodynamics.

The demon haunted physicists for over a century. Could a sufficiently clever being defeat entropy?

The Resolution: Landauer's Principle

The resolution came in stages. In 1961, Rolf Landauer established a fundamental principle connecting information processing to thermodynamics:

At room temperature, $k_B T \ln 2 \approx 2.9 \times 10^{-21}$ J per bit -- incredibly small, but nonzero. Current computers dissipate about $10^6$ times more than this Landauer limit, but the fundamental bound exists regardless of technology.

In 1982, Charles Bennett completed the exorcism of Maxwell's demon. The demon can observe and sort molecules reversibly, without entropy cost. But eventually its memory fills up. When it erases its memory to start over, Landauer's principle guarantees that the erasure produces enough entropy to satisfy the second law. The second law is saved -- not by the act of measurement, but by the necessity of resetting the measuring device.

Jaynes' Maximum Entropy Principle

In the 1950s, Edwin Jaynes proposed a radical reinterpretation of statistical mechanics based on information theory. Instead of deriving the Boltzmann distribution from dynamical assumptions (ergodicity, molecular chaos), Jaynes derived it from a principle of maximum ignorance:

Maximizing $S = -k_B \sum p_i \ln p_i$ subject to the constraint $\sum p_i E_i = \langle E \rangle$ and $\sum p_i = 1$ using Lagrange multipliers immediately gives:

$$p_i = \frac{1}{Z} e^{-\beta E_i}$$

The Boltzmann distribution emerges not from any physical assumption about dynamics, but purely from the requirement of maximal ignorance given what we know. The temperature $T = 1/(k_B \beta)$ appears as a Lagrange multiplier enforcing the energy constraint.

This perspective unifies all the ensembles: the microcanonical ensemble maximizes entropy with a fixed energy constraint; the canonical ensemble adds a temperature constraint; the grand canonical adds a chemical potential constraint. Each ensemble is the maximally ignorant distribution consistent with its constraints.

Quantum Information and von Neumann Entropy

In quantum mechanics, the state of a system is described by a density matrix $\rho$ rather than a probability distribution. The quantum generalization of Shannon entropy is the von Neumann entropy:

$$S_{\text{vN}} = -\text{Tr}(\rho \ln \rho) = -\sum_i \lambda_i \ln \lambda_i$$

where $\lambda_i$ are the eigenvalues of $\rho$. For a pure state, $S_{\text{vN}} = 0$: there is no uncertainty. For a maximally mixed state of dimension $d$, $S_{\text{vN}} = \ln d$: maximum ignorance.

The von Neumann entropy captures a distinctly quantum form of ignorance. When a bipartite system is in an entangled state, the reduced density matrix of each subsystem has nonzero entropy even though the total state may be pure. This entanglement entropy has become a central concept in modern theoretical physics, playing a key role in understanding quantum gravity, black holes, and the structure of spacetime itself.

Black Hole Entropy

The most dramatic connection between information and physics occurs at the intersection of quantum mechanics, gravity, and thermodynamics. In 1972, Jacob Bekenstein proposed that black holes carry entropy proportional to their event horizon area:

This formula is astonishing for several reasons:

It involves all of fundamental physics. The Bekenstein-Hawking formula contains $k_B$ (thermodynamics), $c$ (relativity), $G$ (gravity), and $\hbar$ (quantum mechanics). It stands at the crossroads of our deepest theories.

Entropy scales with area, not volume. For ordinary matter, entropy is extensive -- proportional to volume. Black hole entropy scales with the surface area of the horizon. This suggests that the true degrees of freedom of gravity live on boundaries, not in the bulk. This insight led to the holographic principle: the information content of any region is bounded by the area of its boundary in Planck units.

The numbers are enormous. A solar-mass black hole has $S_{\text{BH}} \sim 10^{77} k_B$, far larger than the entropy of the star that collapsed to form it ($\sim 10^{58} k_B$). Black holes are the most entropic objects in the universe.

The Black Hole Information Paradox

Hawking showed in 1975 that black holes radiate thermally at temperature $T_H = \hbar c^3 / (8\pi G M k_B)$. As the black hole evaporates, what happens to the information that fell in? If the radiation is truly thermal (random), the information is destroyed -- violating quantum mechanics, which demands that information is preserved (unitary evolution). If information is preserved, it must be encoded in subtle correlations in the Hawking radiation -- but how?

This information paradox has driven theoretical physics for half a century. Recent progress, including the "island formula" and insights from the AdS/CFT correspondence, suggests that information is indeed preserved, but the mechanism involves deep connections between quantum entanglement and the geometry of spacetime.

The Thermodynamics of Computation

Landauer's principle connects information processing to physics. What are the ultimate physical limits on computation?

Reversible computation: Bennett showed that any computation can be performed reversibly (without erasing information), incurring no fundamental thermodynamic cost. Only logically irreversible operations (like erasing or overwriting data) must produce heat.

Quantum computation: Quantum mechanics allows fundamentally new types of information processing. Quantum bits (qubits) can exist in superposition and be entangled, enabling algorithms that are exponentially faster for certain problems (like factoring large numbers).

Ultimate limits: The Margolus-Levitin theorem states that the maximum rate of computation is bounded by the energy of the system: $\text{operations/second} \leq 2E/(\pi\hbar)$. The Bekenstein bound limits the information content of a region by its energy and size. Together, these suggest fundamental physical limits to how fast and how densely information can be processed.

The Deep Connection

The information-theoretic perspective reveals a profound unity in physics. Consider how many apparently different concepts reduce to the same mathematical structure:

Domain	Entropy Formula	What It Measures
Statistical mechanics	$S = k_B \ln \Omega$	Number of microstates
Gibbs/canonical	$S = -k_B \sum p_i \ln p_i$	Ignorance of microstate
Information theory	$H = -\sum p_i \log_2 p_i$	Bits needed to encode
Quantum mechanics	$S = -\text{Tr}(\rho \ln \rho)$	Quantum uncertainty
Black holes	$S = A/(4\ell_P^2)$	Horizon information capacity

The fact that the same mathematical structure governs heat engines, communication channels, quantum entanglement, and black holes suggests that information is not merely a useful analogy for entropy -- it may be the most fundamental concept in physics. As John Wheeler put it: "It from bit."

Key Insights

• Shannon entropy $H = -\sum p_i \log p_i$ and Boltzmann-Gibbs entropy are mathematically identical -- both measure ignorance about the state of a system
• Landauer's principle establishes that erasing one bit of information costs at least $k_B T \ln 2$ in energy -- information processing is fundamentally thermodynamic
• Maxwell's demon is defeated by the thermodynamic cost of erasing its memory, not by the act of measurement
• Jaynes' maximum entropy principle derives statistical mechanics from information theory: the Boltzmann distribution maximizes ignorance given constraints
• Black hole entropy is proportional to horizon area, not volume, suggesting the holographic principle
• The black hole information paradox -- whether information is destroyed or preserved during evaporation -- drives research at the frontier of quantum gravity
• Information may be the most fundamental concept in physics, underlying thermodynamics, quantum mechanics, and gravity

Looking Ahead

With statistical physics complete, we have the tools to understand matter in all its states and the deep connections between information, entropy, and the physical world. The journey continues with general relativity -- Einstein's theory of gravity as curved spacetime:

• Differential geometry -- the mathematical language of curved spaces
• Einstein's field equations -- how matter tells spacetime to curve
• Black holes and gravitational waves -- where the information-theoretic ideas of this lesson meet the geometry of spacetime

The themes of entropy, information, and symmetry will follow us all the way to the frontiers of physics.