Information theory
Information theory is the mathematical study of quantifying information and uncertainty, and of representing, encoding, storing, processing, and communicating information.
Bayesian Optimization in Action
Information theory is the mathematical study of information, uncertainty, and the relationships between probability distributions and observations.
- How it is used
- Guides Bayesian optimization decisions toward evaluations expected to reduce uncertainty about an objective function or global optimum.
- How deep it goes
- Defines entropy, computes it for a uniform search distribution, and derives binary search through expected posterior entropy.
- Examples
- Uses remote-control search, a 100-element binary search, and a prior-informed phone-drop threshold exercise.
- What you can do after
- Apply it to choose searches or evaluations that minimize expected posterior uncertainty under a specified prior.
Read how Bayesian Optimization in Action explains information theory →
How Large Language Models Work
The mathematical study of information as a quantifiable resource, including its representation, transmission, and uncertainty.
- How it is used
- Uses information as a measurable resource to assess claims that language models can improve themselves through repeated self-training.
- How deep it goes
- Defines information distribution and noisy reproduction, but stops short of mathematical derivations or formal measures.
- Examples
- Uses an iterative loop of generating data, retraining, and observing degraded quality to illustrate information loss.
- What you can do after
- Evaluate/understand why synthetic data alone cannot supply new information for indefinite model improvement.
Read how How Large Language Models Work explains information theory →
Related concepts
- informationmeasures · Information is content that conveys meaningful distinctions or reduces uncertainty about possible states, observations, or outcomes.
- context assemblyuses · Context assembly is the process of selecting, organizing, and combining relevant information and instructions into context for a model or other downstream processing.
- entropyuses · Entropy is a quantitative measure of uncertainty or dispersion in a probability distribution and, more generally, of disorder in the possible states of a system.
- binary searchuses · Binary search is an algorithm that locates a target or boundary in an ordered collection by repeatedly comparing with a midpoint and halving the remaining search range.
- Bayesian optimizationuses · Bayesian optimization is a sequential method for optimizing an expensive or unknown objective by using a probabilistic model to guide evaluations.
- max-value entropy searchused by · Max-value Entropy Search is a Bayesian optimization acquisition policy that selects evaluations expected to reduce uncertainty about the global maximum value of an objective.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.