CONCEPT
Entropy (Information-Theoretic)
Shannon's measure of the average surprise per message from a source — high entropy means unpredictable messages carrying genuine information, low entropy means predictable messages carrying almost none.
In Shannon's framework, entropy is not disorder but surprise. A source producing predictable messages has low entropy; a source producing unpredictable messages has high entropy. The information content of any single message is inversely proportional to its probability: a weather forecast of 'sunny' in the Sahara carries almost zero bits, while 'volcanic ash advisory' carries many. This counterintuitive definition — that information is what you did not expect — reframes the human-AI collaboration. The value of an exchange is bounded by the entropy of the question. Narrow, predictable queries produce low-information retrieval; broad, uncertain questions produce high-information synthesis. The quality of your curiosity determines the upper bound on what any amplifier can deliver.
In The You On AI Field Guide
Shannon's formal definition, H = -Σ p(x) log₂ p(x), measures the average number of bits needed to encode a message from a given probability distribution. The logarithmic form ensures that independent sources combine additively — a cornerstone property that makes the measure useful across