liveBook · Manning

Sparsity

As explained in Machines that Think

Sparsity is the condition in which most possible sequences or combinations are absent or too infrequent in observed data for their probabilities to be estimated reliably.

Sparsity arises when most possible word sequences are absent or too rare in observed data to estimate reliably. In n-gram models, an unseen sequence can receive zero probability, making a sentence containing it impossible for the model to recognize or produce. section 2.3.4

Smoothing addresses this failure by giving unseen sequences a small probability. Laplace smoothing assigns probability to every possible sequence; backoff models use shorter contexts when longer ones are too sparse. Longer n-grams capture more context, but their many possible combinations require more data and make reliable estimates harder. section 2.3.4 section 2.3.5

A phrase the model has not seen

A trigram model may have many examples of “in the world” but none of “quantum donut theory.” Backoff can estimate the unfamiliar phrase using shorter fragments found elsewhere in the data. section 2.3.4

Figure 2.4 Balancing context and complexity in n-gram models. As the length of word sequences increases, models capture richer context but require substantially more data to estimate probabilities reliably. Longer sequences allow more precise patterns to be represented, but they also introduce a rapid growth in the number of possible combinations, making reliable estimation increasingly difficult.

Compare how longer n-grams represent richer context while multiplying possible combinations and increasing the data needed for reliable estimates. section 2.3.5

How Machines that Think covers it

How deep it goes
Defines sparsity, traces zero-probability failure, and outlines smoothing approaches without detailing their algorithms.
Examples
Contrasts common “in the world” with unseen “quantum donut theory” and illustrates fallback to shorter fragments.
What you can do after
Evaluate n-gram context depth against data needs and understand why smoothing helps with unseen sequences.
How it connects
Connects sparsity to n-gram models, smoothing, Laplace smoothing, and backoff models.

Also explained in

Compare how 6 books explain sparsity →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Machines that Think. The section links point to the parts of the book it draws on.