Statistical generation predicts and produces text by learning probability distributions over sequences. For a name such as “John,” the model estimates the likelihood of J, followed by o, h, and n in order. section 9.2
A simple statistical generator works by:
- Collecting a large dataset, such as 31,915 names from the 2022 U.S. baby-name list. section 9.2
- Counting consecutive character pairs and storing their frequencies in a 27 × 27 matrix. section 9.2
- Normalizing the counts into probabilities, so each row estimates the next character given the current character. section 9.2
- Sampling repeatedly from those probabilities until an end-of-name marker appears. section 9.2
Generating names with a bigram model
Represent each name with start and end markers, such as "$john$". The model samples a character after "$", then after that character, and continues until it samples another "$". Because it learns only adjacent-character patterns, outputs such as “liahuli,” “deiannnis,” and “neynee” can resemble names without reproducing the training examples. section 9.2
# In[]:
bigram = torch.zeros((vocab_size, vocab_size)) #1
total = 0
for name in names:
for ch1, ch2 in zip(name, name[1:]):
ch1_int = ch_to_i[ch1]
ch2_int = ch_to_i[ch2]
bigram[ch1_int][ch2_int] += 1
total += 1
bigram /= total
Inspect the PyTorch listing that counts consecutive characters and normalizes the resulting bigram matrix. section 9.2
Bigram generation is deliberately crude: it considers only the previous character. More sophisticated language models use tokens and learned sequence structure; embeddings, linear layers, and attention support richer generation, while decoder-only transformers generate from broader context. section 9.12
