A bigram model predicts the next character from the character immediately before it. It learns pairwise transition probabilities by counting adjacent characters in a collection of names, making statistical text generation concrete before introducing more sophisticated neural models. section 9.2
# In[]:
names = []
with open('../data/p2ch9/names_2022.txt', 'r') as file:
for line in file:
name, _, _= line.lower().strip().split(',')
names.append("$" + name + "$")
len(names)
# Out[]:
31915
The dataset setup wraps each lowercase name with boundary markers, producing 31,915 training examples for character-pair counting. section 9.2
A simple implementation follows three steps:
- Represent the transitions as a 27 × 27 matrix: each row identifies the preceding character, and each column identifies a possible next character. section 9.2
- Count every consecutive character pair across the names, then normalize the matrix so its entries represent probabilities. section 9.2
- Generate a name by sampling from the row associated with the latest character, appending the sampled character, and stopping at the boundary marker. section 9.2
Sampling generated names
Starting from "$", repeatedly sample the next character from its learned probability row. Outputs such as “liahuli,” “deiannnis,” “cl,” “de,” and “neynee” demonstrate both recognizable fragments and crude results. section 9.2
The model captures only one-character context, so it crudely estimates sequences compared with models using embeddings, linear layers, attention, or transformer architectures. A useful extension asks how name frequency would change the model. section 9.11 section 9.12
