Information theory
Information theory is the mathematical study of information, uncertainty, and the quantification, encoding, storage, processing, and communication of information.
Bayesian Optimization in Action
Information theory is the mathematical study of information, uncertainty, and the relationships between probability distributions and observations.
- How it is used
- Guides Bayesian optimization decisions toward evaluations expected to reduce uncertainty about an objective function or global optimum.
- How deep it goes
- Defines entropy, computes it for a uniform search distribution, and derives binary search through expected posterior entropy.
- Examples
- Uses remote-control search, a 100-element binary search, and a prior-informed phone-drop threshold exercise.
- What you can do after
- Apply it to choose searches or evaluations that minimize expected posterior uncertainty under a specified prior.
Read how Bayesian Optimization in Action explains information theory →
How Large Language Models Work
The mathematical study of information as a quantifiable resource, including its representation, transmission, and uncertainty.
- How it is used
- Uses information as a measurable resource to assess claims that language models can improve themselves through repeated self-training.
- How deep it goes
- Defines information distribution and noisy reproduction, but stops short of mathematical derivations or formal measures.
- Examples
- Uses an iterative loop of generating data, retraining, and observing degraded quality to illustrate information loss.
- What you can do after
- Evaluate/understand why synthetic data alone cannot supply new information for indefinite model improvement.
Read how How Large Language Models Work explains information theory →
Timeless Algorithms, The Foundational Ideas
Information theory is the mathematical study of information, uncertainty, and communication, including measures of entropy and error.
- How it is used
- Interprets cross-entropy as an information cost for observed outcomes in classification.
- How deep it goes
- Defines event information as negative log probability and relates binary cross-entropy to likelihood.
- Examples
- Illustrates event information with probabilities 0.68 and 0.99, yielding costs 0.39 and 0.01.
- What you can do after
- Understand why cross-entropy penalizes low probability assigned to observed classes and serves as a classification objective.
Read how Timeless Algorithms, The Foundational Ideas explains information theory →
Also covered in
- Context EngineeringIn early access (MEAP)
What to know first
- objective functionAn objective function is a mathematical function that assigns values to candidate solutions or parameter settings, which an optimization procedure seeks to minimize or maximize.
Related concepts
- entropyhas part · Entropy is an information-theoretic measure of uncertainty in a probability distribution, defined as the expected value of the negative logarithm of outcome probability.
- informationmeasures · Information is content that conveys meaningful distinctions or reduces uncertainty about possible states, observations, or outcomes.
- context assemblyuses · Context assembly is the process of selecting, organizing, and combining relevant information and instructions into context for a model or other downstream processing.
- Bayesian optimizationuses · Bayesian optimization is a sequential method for optimizing an expensive or unknown objective by using a probabilistic model to guide evaluations.
- binary searchuses · Binary search is an algorithm that repeatedly halves an ordered search interval to locate a target or determine its position or boundary.
- max-value entropy searchused by · Max-value Entropy Search is a Bayesian optimization acquisition policy that selects evaluations expected to reduce uncertainty about the global maximum value of an objective.
Questions about information theory
What is information theory?
Information theory is the mathematical study of information, uncertainty, and the quantification, encoding, storage, processing, and communication of information.
Which Manning books explain information theory?
Bayesian Optimization in Action, How Large Language Models Work and Timeless Algorithms, The Foundational Ideas.
What should I know before learning information theory?
Manning books that teach information theory usually introduce objective function first.
What are the parts of information theory?
Information theory includes entropy.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.


