liveBook · Manning

Bit

As explained in Timeless Algorithms, The Foundational Ideas

A bit is a binary unit of information that can take one of two possible values.

A bit, or binary digit, is the unit for expressing entropy. It represents uncertainty in a choice between two equally likely alternatives: information is measured by the uncertainty it resolves, not by meaning or message length. section 5.1.3 section 5.0

H(X)=-[(0.5)log2(0.5)+(0.5)log2(0.5)]

For a fair coin, the two outcomes are equally likely; the entropy calculation gives one bit of uncertainty before the outcome is observed. section 5.1.3

More outcomes or less predictable outcomes require more bits on average. This common scale makes different information sources comparable and helps quantify information and redundancy in communication. section 5.1.3 section 5.6

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Defines a bit through equally likely binary choices and calculates fair-coin entropy, building on entropy as a measure of uncertainty.
Examples
Uses fair and biased coins, a six-sided die, and a standard deck to illustrate entropy measured in bits.
What it assumes
Builds on entropy as a measure of uncertainty while defining the bit as its unit.
What you can do after
Understand how to express an information source’s uncertainty in bits and compare sources using that measure.
How it connects
Links bits to entropy, uncertainty, information theory, communication, compression, and machine learning.

Also explained in

Compare how 6 books explain bit →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.