A bigram is a pair of adjacent words or tokens. In language prediction, it estimates how likely one item is to follow another by counting each previous–next pairing and dividing by all occurrences of the previous item. section 11.2 section 11.9
Bigrams support two closely related tasks:
- Next-word prediction: use pairs such as “Alice was” or “White Rabbit” to build a probability table for autocomplete-style completions. section 11.2
- Tokenization: count neighboring character or subword tokens, then merge the most frequent pair repeatedly as part of byte-pair encoding (BPE). section 11.9
Predicting the next word
Across five sample sentences, “Alice” is followed by “was” twice and “and” once. The model therefore assigns probabilities of 0.67 and 0.33, making “Alice was” the more likely completion. Likewise, “the” leads to “White” twice and “Rabbit” once. section 11.2
