liveBook · Manning

Language model

As explained in Grokking Bayes

A language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.

A large language model (LLM) predicts what word might come next in a sentence. Because several words may fit, it assigns different probabilities to possible continuations: after “Bayesian probability,” “theory” or “examples” are plausible, while “cat” is not. section 1.7

The prediction depends on both the preceding words and the data used to train the model. Conditioning on different starting phrases lets an LLM account for different contexts. Bayesian probability offers one way to relate these conditional probabilities to a prior, data, and posterior; LLMs can be Bayesian in spirit without that making them straightforwardly Bayesian models. section 1.7

Pr(… | “Bayesian probability”, 𝐷),

This expression represents the probability of a continuation given the phrase “Bayesian probability” and training data D. section 1.7

How Grokking Bayes covers it

How deep it goes
Explains probabilistic next-word prediction and contextual conditioning, then defers the calculation of conditional probabilities.
Examples
Uses possible continuations of “Bayesian probability” to contrast plausible predictions with an unlikely one.
What you can do after
Understand how next-word prediction uses probabilities and context, and how this relates to Bayesian thinking.
How it connects
Connects language models to conditional probability, training data, and Bayesian concepts including priors and posteriors.

Also explained in

Compare how 89 books explain language model →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Grokking Bayes. The section links point to the parts of the book it draws on.