Entropy (Information-Theoretic)

In the AI Story

Shannon's formal definition, H = -Σ p(x) log₂ p(x), measures the average number of bits needed to encode a message from a given probability distribution. The logarithmic form ensures that independent sources combine additively — a cornerstone property that makes the measure useful across domains.

Applied to question engineering, entropy becomes a quality metric for prompts. The question 'What is the capital of France?' specifies a distribution with nearly all probability mass on one answer — entropy near zero, information near zero. The question 'What should we build?' distributes probability across an enormous answer space — entropy high, information high. The exchange's value is bounded by the entropy the question opens.

The distinction illuminates the confusion between throughput and information in AI discourse. A system generating thousands of predictable documents is operating at high throughput and zero entropy. A system producing a single unexpected synthesis is operating at low throughput and high information. These quantities are independent, and most AI productivity metrics measure the wrong one.

The framework maps directly onto Segal's distinction between questions and answers. Questions diverge (high entropy); answers converge (low entropy). The scarce, valuable, irreplaceable human contribution is the generation of high-entropy questions — the capacity to open spaces whose resolution cannot be predicted.

Origin

Shannon borrowed the term entropy from statistical mechanics, reportedly on the suggestion of John von Neumann, who told him: 'Call it entropy. Nobody knows what entropy really is, so in a debate you will always have the advantage.' The mathematical analogy to Boltzmann entropy proved deeper than a mere naming convenience.

Key Ideas

Information is surprise. The information content of a message is inversely proportional to its probability — certainty carries zero bits.

Entropy bounds information. The average information from a source cannot exceed its entropy rate.

Question entropy bounds exchange value. The information-theoretic value of any query-response pair is bounded above by the entropy of the query.

Throughput and information are independent. A high-volume output can carry negligible information if its entropy is low.

High-entropy curiosity is scarce. The capacity to ask questions whose answers cannot be predicted is the binding constraint on AI collaboration's value.