How deeply does How Large Language Models Work cover information theory?

Defines information distribution and noisy reproduction, but stops short of mathematical derivations or formal measures.

Information theory

As explained in How Large Language Models Work

The mathematical study of information as a quantifiable resource, including its representation, transmission, and uncertainty.

Information theory treats information as a quantifiable resource. Applied to language models, it frames training as approximating the information distribution in an original dataset rather than creating information from nothing. section 7.1.1

Why repeated self-training degrades

Suppose a model generates data, adds it to its training set, and trains again. Each generated sample is a noisy, incomplete reproduction of the original distribution, so repeated rounds cannot add information absent from the original data and can reduce quality. Improvement requires external or genuinely new information at each round. section 7.1.1

How How Large Language Models Work covers it

liveBook manning.com

Read How Large Language Models Work on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

Bayesian Optimization in ActionInformation theory is the mathematical study of information, uncertainty, and the relationships between probability distributions and observations.

Compare how 2 books explain information theory →

Related concepts

informationmeasures · Information is content that conveys meaningful distinctions or reduces uncertainty about possible states, observations, or outcomes.
context assemblyuses · Context assembly is the process of selecting, organizing, and combining relevant information and instructions into context for a model or other downstream processing.
entropyuses · Entropy is a quantitative measure of uncertainty or dispersion in a probability distribution and, more generally, of disorder in the possible states of a system.
binary searchuses · Binary search is an algorithm that locates a target or boundary in an ordered collection by repeatedly comparing with a midpoint and halving the remaining search range.
Bayesian optimizationuses · Bayesian optimization is a sequential method for optimizing an expensive or unknown objective by using a probabilistic model to guide evaluations.
max-value entropy searchused by · Max-value Entropy Search is a Bayesian optimization acquisition policy that selects evaluations expected to reduce uncertainty about the global maximum value of an objective.

This explainer was generated by Manning's knowledge graph from the text of How Large Language Models Work. The section links point to the parts of the book it draws on.