N-gram model
An n-gram model is a probabilistic language model that estimates sequence probabilities using contiguous sequences of a fixed number of tokens, typically to predict a token from preceding context.
Machines that Think
A language model that estimates the probability of a token from a fixed-length sequence of preceding tokens.
- How it is used
- Introduces statistical language modeling through local frequency-based prediction and its applications in language technology.
- How deep it goes
- Explains conditional-frequency estimates, sparsity, smoothing, and context-size trade-offs without implementing a model.
- Examples
- Calculates the probability of “east” after “in the” from corpus counts and uses an unseen phrase to illustrate sparsity.
- What you can do after
- Evaluates how context length and sparse data affect the reliability and feasibility of n-gram predictions.
LLMs in Production
An n-gram model is a probabilistic language model that estimates word sequences using a fixed-length context of n tokens.
- How it is used
- Provides an efficient generative baseline for modeling local word sequences and comparing more capable language models.
- How deep it goes
- Explains training, generation, scoring, and evaluation, but stops short of modeling syntax or broader linguistic knowledge.
- Examples
- Uses a Hamlet corpus, padded trigrams, generated continuations, token counts, probability scores, entropy, and perplexity.
- What you can do after
- Build it from scratch with Python and NLTK, generate continuations, inspect probabilities, and evaluate phrases with entropy or perplexity.
Also covered in
What to know first
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- perplexityPerplexity is a language-model metric, typically the exponential of average cross-entropy, that measures how well a model predicts a sequence, with lower values indicating better predictive fit.
- speech recognitionSpeech recognition is the computational task of converting spoken language in an audio signal into a machine-readable representation, typically text.
- sequenceA sequence is an ordered collection of elements whose positions determine the order in which they are arranged or traversed.
Related concepts
- bag-of-words modeladdressed by · A bag-of-words model represents text using word or token occurrences, counts, or weights while disregarding their order and syntactic relationships.
- machine translationuses · Machine translation is the automated computational conversion of text or speech from one natural language into another.
- recurrent neural networkcontrasts with · A recurrent neural network is a neural network that processes ordered inputs while carrying and updating an internal state across successive steps.
- speech recognitionuses · Speech recognition is the computational task of converting spoken language in an audio signal into a machine-readable representation, typically text.
- trigram modelincludes · A trigram model represents or predicts sequences using groups of three consecutive units, such as characters or tokens.
Questions about n-gram model
What is n-gram model?
An n-gram model is a probabilistic language model that estimates sequence probabilities using contiguous sequences of a fixed number of tokens, typically to predict a token from preceding context.
Which Manning books explain n-gram model?
Machines that Think and LLMs in Production.
What should I know before learning n-gram model?
Manning books that teach n-gram model usually introduce language model, perplexity, speech recognition and sequence first.
What is the difference between n-gram model and recurrent neural network?
An n-gram model is a probabilistic language model that estimates sequence probabilities using contiguous sequences of a fixed number of tokens, typically to predict a token from preceding context. A recurrent neural network is a neural network that processes ordered inputs while carrying and updating an internal state across successive steps.
What kinds of n-gram model are there?
Kinds of n-gram model include trigram model.
What is n-gram model a kind of?
N-gram model is a kind of model. A model is an abstract representation of a system, phenomenon, application, or domain that captures relevant structure, behavior, data, or beliefs.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.

