Sparsity arises when most possible word sequences are absent or too rare in observed data to estimate reliably. In n-gram models, an unseen sequence can receive zero probability, making a sentence containing it impossible for the model to recognize or produce. section 2.3.4
Smoothing addresses this failure by giving unseen sequences a small probability. Laplace smoothing assigns probability to every possible sequence; backoff models use shorter contexts when longer ones are too sparse. Longer n-grams capture more context, but their many possible combinations require more data and make reliable estimates harder. section 2.3.4 section 2.3.5
A phrase the model has not seen
A trigram model may have many examples of “in the world” but none of “quantum donut theory.” Backoff can estimate the unfamiliar phrase using shorter fragments found elsewhere in the data. section 2.3.4
Figure 2.4 Balancing context and complexity in n-gram models. As the length of word sequences increases, models capture richer context but require substantially more data to estimate probabilities reliably. Longer sequences allow more precise patterns to be represented, but they also introduce a rapid growth in the number of possible combinations, making reliable estimation increasingly difficult.

Compare how longer n-grams represent richer context while multiplying possible combinations and increasing the data needed for reliable estimates. section 2.3.5





