Information entropy and thermodynamic entropy

Specify the possibilities being counted

Shannon entropy H measures uncertainty over a specified collection of alternatives. With natural logarithms it is expressed in nats; with base-two logarithms it is expressed in bits. A distribution over letters, detector outcomes or microscopic states can all have the same numerical H while describing very different systems.

Boltzmann entropy associates a macrostate with the number or phase-space measure of compatible microstates: S_B = k_B ln Ω, with the physical measure and constraints specified. Gibbs entropy uses a distribution over microscopic states. These constructions can reproduce thermodynamic entropy in appropriate statistical descriptions, but their roles should be stated.

A distribution over the two possible words “hot” and “cold” is not a complete microscopic ensemble for the object being labelled. Multiplying its uncertainty by k_B does not turn it into that object's thermodynamic entropy. The missing physical identification cannot be supplied by changing units.

Make the connection with an energy model

Consider a two-level system with energies zero and ε, in canonical equilibrium at temperature T. Assume equal degeneracies and define β = 1/(k_BT). The probabilities are π₀ = 1/Z and π₁ = e⁻ᵝᵋ/Z, where Z = 1 + e⁻ᵝᵋ.

Because ln π_i = −βE_i − ln Z, direct substitution gives S = −k_BΣ π_i ln π_i = k_B ln Z + U/T. The energy levels, temperature and equilibrium preparation connect the probability formula to thermodynamic quantities.

Choose βε = ln 3. Then π₁ = 1/4, π₀ = 3/4 and S/k_B = 0.562335. The equilibrium distribution is not uniform. Its uncertainty is less than ln 2, even though both states remain possible.

Relaxation can lower the system's entropy

Prepare the same fixed-energy system with probabilities q = (1/2,1/2), then let it relax to π in contact with that bath, with no work performed during relaxation. Initially its dimensionless entropy is ln 2 ≈ 0.693147. Its final entropy is lower by approximately 0.130812 k_B.

The mean energy decreases from ε/2 to ε/4, so the bath receives ε/4. Its entropy increase is ε/(4T) = k_B ln 3/4 ≈ 0.274653 k_B. The combined entropy increase is therefore approximately 0.143841 k_B, which is positive.

The subsystem entropy decrease is not a violation of the second law. It is part of a complete balance. Nor should the equilibrium state's smaller Shannon entropy be described as less physical equilibrium merely because it is farther from a uniform distribution.

Relative entropy selects the relevant reference

Define D(q‖π) = Σ q_i ln(q_i/π_i). For this preparation, D(q‖π) = ½ ln(2/3) + ½ ln 2 ≈ 0.143841. It matches the total entropy change in units of k_B for the specified complete relaxation.

For a distribution q on these fixed energy levels, define the nonequilibrium free-energy functional F[q] = Σ q_iE_i − T S[q]. Algebra gives F[q] − F[π] = k_BT D(q‖π). The canonical reference carries the temperature and energy information that a comparison with the uniform distribution would omit.

A suitable evolution may contract D relative to a stationary state without increasing S itself. That is why the identity of the stationary state and the physical meaning of heat exchange matter when an entropy theorem is interpreted.

Choose the entropy before claiming its arrow

The expanded book's physical-reduction argument distinguishes relative-entropy contraction from unconditional subsystem entropy growth. Its dephasing example concerns another regime, where populations remain fixed while reduced coherence changes. Different observables reveal different parts of the evolution.

For SMM/DTT, a constructive next step is to derive the completed state space, its physical equilibrium reference and the exchanged quantities from one specified model. An entropy functional then has a clear target. Assigning the same name to record uncertainty, thermodynamic entropy and an increasing event counter would conceal the needed connections.

Sources and further reading

• David Tong, canonical ensembles, entropy and free energy.
• H. Spohn, Entropy production for quantum dynamical semigroups, Journal of Mathematical Physics 19, 1227–1230 (1978), for a stationary-state entropy-production framework.
• Continue with open-system balances.