Compression represents information with fewer bits by removing predictable redundancy without changing the information to be recovered. Entropy measures source uncertainty: AAAAAA can be written A × 6, while W7dX8$ offers little redundancy to remove. In communication systems, this distinction helps estimate transmission needs and channel capacity. Redundancy is not always waste: predictable redundancy can be removed, while deliberately added redundancy can help detect or correct errors caused by noise. section 5.1.4
Figure 5.1 Entropy measures the uncertainty associated with an information source in bits. A fair coin has an entropy of 1 bit, while a biased coin has lower entropy (0.47 bits) because one outcome is much more predictable than the other. A fair die (2.59 bits) and a standard deck of cards (5.7 bits) have progressively higher entropy because they contain larger sets of equally likely outcomes. As entropy increases, preserving the original information becomes increasingly important because there is less redundancy available for compression. Conversely, as entropy decreases, greater redundancy creates more opportunities to compress the source without losing information.

Compare entropy across coins, dice, and cards to see how greater uncertainty leaves less redundancy available for compression. section 5.1.4














