Square h5 · Physics · Kingside

Shannon

Shannon entropy of a discrete distribution.

Meaning

Shannon entropy H=−∑ipilog⁡piH=-\sum_i p_i\log p_i measures the average information content, or uncertainty, of a discrete probability distribution {pi}\{p_i\}. When the logarithm is base 2, HH is in bits; with natural log it is in nats. Certain outcomes (p=1p=1) contribute zero; a fair coin yields H=1H=1 bit; a uniform distribution over NN symbols yields log⁡N\log N, the maximum for that alphabet. The formula is concave, extensive for independent sources, and the unique (up to scaling) functional satisfying Shannon’s natural axioms for information.

It is the communication-theoretic cousin of Boltzmann–Gibbs entropy S=−k∑piln⁡piS=-k\sum p_i\ln p_i: the mathematics coincides, the interpretation shifts from thermodynamic multiplicity to coding length. Source coding theorems say that HH is the asymptotic lower bound on average lossless compressed length.

Context

Claude Shannon defined this entropy in his 1948 paper “A Mathematical Theory of Communication,” founding information theory. The work quantified channel capacity, introduced the bit as a unit, and separated information from meaning—a move that transformed engineering, cryptography, and statistical inference. Jaynes later championed the formal parallel with statistical mechanics, while Kolmogorov and others developed algorithmic information alternatives.

On a physics board, Shannon’s HH earns its square because modern thermodynamics, quantum information (S=−Tr ρlog⁡ρS=-\mathrm{Tr}\,\rho\log\rho), and black-hole entropy debates all speak its dialect. Whether one is designing an error-correcting code or arguing about Landauer’s principle, the sum −∑pilog⁡pi-\sum p_i\log p_i is the starting measure of ignorance and surprise.

Relative entropy (Kullback–Leibler divergence) and mutual information extend Shannon’s HH to comparisons between distributions and to shared information between variables. Those descendants power modern machine learning objectives and statistical-mechanics proofs of the second law’s information form.

Sources