Entropy & the Second Law

We now enter statistical physics, the framework that connects the microscopic world of atoms to the macroscopic world of temperature, pressure, and heat. The central concept is entropy, arguably the most profound idea in all of physics. It explains why ice melts, why engines have limited efficiency, why we age, and why time appears to flow in one direction.

The remarkable insight of statistical mechanics is that all of thermodynamics can be derived from one simple idea: count the number of microscopic arrangements consistent with what we observe macroscopically.

Microstates and Macrostates

A microstate is a complete specification of every microscopic degree of freedom: the position and momentum of every particle, or the quantum state of every component. For a box of gas with $N \sim 10^{23}$ molecules, a microstate specifies $\sim 6 \times 10^{23}$ numbers.

A macrostate is a description in terms of a few macroscopic variables: total energy $E$, volume $V$, number of particles $N$, temperature $T$, pressure $P$. Many different microstates correspond to the same macrostate.

The number of microstates corresponding to a given macrostate is called the multiplicity, denoted $\Omega$. This single number is the key to everything.

A Concrete Example: Coin Flipping

Consider 100 coins. The macrostate "all heads" has exactly $\Omega = 1$ microstate. The macrostate "50 heads, 50 tails" has $\Omega = \binom{100}{50} \approx 10^{29}$ microstates. That's the ratio $10^{29}$ to $1$. The disordered state is overwhelmingly more likely, not because of any force, but simply because of counting.

Now imagine $10^{23}$ coins. The dominance of the most probable macrostate becomes so extreme that deviations from it are never observed in practice. This is the essence of the second law.

Boltzmann's Entropy

In 1877, Ludwig Boltzmann made the profound connection between microscopic counting and the macroscopic quantity called entropy:

Why the logarithm? Two key reasons:

Additivity. If two independent systems have multiplicities $\Omega_1$ and $\Omega_2$, the combined system has $\Omega = \Omega_1 \cdot \Omega_2$. Taking the logarithm gives $S = S_1 + S_2$, making entropy additive, an extensive quantity, as thermodynamics requires.

Manageability. For macroscopic systems, $\Omega$ is absurdly large (like $10^{10^{23}}$). The logarithm brings this down to a human-scale number.

Entropy of the Ideal Gas

For an ideal gas of $N$ particles in volume $V$ with total energy $E$, counting the accessible phase space volume yields the Sackur-Tetrode equation:

$$S = Nk_B \left[\ln\left(\frac{V}{N}\left(\frac{4\pi m E}{3Nh^2}\right)^{3/2}\right) + \frac{5}{2}\right]$$

This matches all classical thermodynamic results. Entropy increases with energy, volume, and particle number, exactly as expected. Boltzmann's formula is not just a definition; it reproduces all of thermodynamics from first principles.

The Second Law of Thermodynamics

The second law is traditionally stated in many equivalent forms. In statistical mechanics, it reduces to a statement about probability:

This is not a fundamental law imposed from outside. It follows from the assumption that all accessible microstates are equally probable (the fundamental postulate of statistical mechanics) and the overwhelming dominance of high-entropy macrostates.

Why Entropy Increases: The Tyranny of Large Numbers

Consider a gas initially confined to the left half of a box. When the partition is removed, why does it expand to fill the whole box? Each molecule independently has a 50% chance of being in either half. The probability that all $N$ molecules remain in the left half is:

$$P = \left(\frac{1}{2}\right)^N$$

For $N = 10^{23}$, this probability is $P \sim 10^{-3 \times 10^{22}}$: a number so unimaginably small that it will never happen in the lifetime of the universe. The gas expands not because it is forced to, but because the expanded state has astronomically more microstates.

Irreversibility from Reversible Laws

Here lies one of the deepest puzzles in physics. The microscopic laws: Newton's equations, the Schrödinger equation, are time-reversible. If you reverse all velocities, the system retraces its path. Yet macroscopic behavior is clearly irreversible: eggs break but don't unbreak, perfume diffuses but doesn't spontaneously concentrate.

How does irreversibility arise from reversible laws? The answer is not dynamical but statistical. The reversed motion is a valid solution of the equations, but it corresponds to an initial condition of extraordinary precision, one that is overwhelmingly unlikely to occur spontaneously.

Loschmidt's Paradox

Loschmidt objected to Boltzmann in 1876: if the laws are time-reversible, then for every entropy-increasing trajectory there should be an entropy-decreasing one. Boltzmann's response was profound: yes, the reversed trajectories exist, but they require initial conditions of exquisite fine-tuning. Starting from a "typical" initial condition, the system will almost certainly evolve toward higher entropy.

Boltzmann's H-Theorem

Boltzmann tried to prove the second law mechanically. He defined the $H$-function for a gas:

$$H = \int f(\mathbf{v}) \ln f(\mathbf{v}) \, d^3v$$

where $f(\mathbf{v})$ is the velocity distribution function. Using the Boltzmann equation (which includes molecular collisions), he showed:

$$\frac{dH}{dt} \leq 0$$

Since $S = -Nk_B H$ (up to constants), this means entropy increases.

The key assumption is molecular chaos (Stoßzahlansatz): before any collision, the velocities of the colliding particles are uncorrelated. This is where the time asymmetry enters, it's an assumption about initial conditions, not about the dynamical laws themselves.

At equilibrium, $dH/dt = 0$, and the distribution that minimizes $H$ is the Maxwell-Boltzmann distribution:

$$f(\mathbf{v}) = n\left(\frac{m}{2\pi k_B T}\right)^{3/2} \exp\left(-\frac{mv^2}{2k_B T}\right)$$

The Arrow of Time

The second law provides the thermodynamic arrow of time: the future is the direction in which entropy increases. But why was entropy low in the past? This question pushes us beyond thermodynamics into cosmology.

The early universe, just after the Big Bang, was in a state of remarkably low gravitational entropy: matter was spread almost uniformly. As gravity clumped matter into stars, galaxies, and black holes, gravitational entropy increased enormously. Roger Penrose has argued that explaining this initial low entropy, the "Past Hypothesis", is one of the deepest unsolved problems in physics.

Temperature from Entropy

Statistical mechanics gives a microscopic definition of temperature. If two systems can exchange energy, equilibrium occurs when entropy is maximized. This gives:

$$\frac{1}{T} = \frac{\partial S}{\partial E}\bigg|_{V,N}$$

Temperature measures how entropy changes with energy. A "hot" system has slowly changing entropy (adding energy doesn't create many new microstates), while a "cold" system has rapidly changing entropy. Energy flows from hot to cold because the total entropy increases when it does.

Connection to Information Theory

In 1948, Claude Shannon discovered that the formula for information entropy is identical to Boltzmann's formula:

$$S_{\text{Shannon}} = -k \sum_i p_i \ln p_i$$

This is not a coincidence. Entropy measures our ignorance about which microstate the system is in. High entropy means many possible microstates, maximum uncertainty. Low entropy means few microstates, we know more about the system. This deep connection between entropy and information will be explored fully in Lesson 5.5.

Key Insights

• Entropy $S = k_B \ln \Omega$ counts the number of microstates compatible with a macrostate
• The second law is a consequence of probability: systems evolve toward macrostates with more microstates
• Irreversibility emerges from reversible laws because high-entropy states vastly outnumber low-entropy ones
• Boltzmann's H-theorem shows entropy increase follows from molecular chaos
• The arrow of time traces back to the low-entropy initial conditions of the universe
• Temperature is defined by how entropy changes with energy: $1/T = \partial S / \partial E$
• Entropy is deeply connected to information, it measures what we don't know about a system

Looking Ahead

We've established the foundational concepts of statistical mechanics. In the next lesson, we'll build the practical computational machinery:

• Ensembles: systematic frameworks for calculating macroscopic properties from microscopic physics
• The partition function: a single mathematical object from which all thermodynamics follows
• Free energies: the thermodynamic potentials that govern systems in contact with reservoirs

The counting of microstates will become the partition function, the most powerful tool in statistical physics.