Defines the set of unique characters or tokens used by a sequence model. A dictionary assigns each item an integer ID, producing a tensor representation that PyTorch can process; Σ denotes the vocabulary and |Σ| its size. section 4.2.1 section 4.2.2
Builds and uses a vocabulary through a small, explicit pipeline:
- Choose normalization deliberately. Lowercasing reduces the vocabulary and task difficulty, but removes capitalization information. section 7.5.1
- Scan the corpus and assign each unseen character or token the next integer, then construct the reverse integer-to-item mapping. section 7.5.1
- Convert each sequence into torch.long IDs while preserving item order; the vocabulary size becomes the number of possible model symbols. section 4.2.1 section 4.2.2
- Add control symbols when sequences need structure: PAD marks unused positions, SOS starts a source sequence, and EOS marks a completed target. section 11.2.1
Character-level Shakespeare model
Scanning the lowercased corpus produces 36 unique characters from 99,993 characters. A chunk such as "hello" becomes integer IDs, and an nn.Embedding layer maps those IDs to vectors before the recurrent model processes them. section 7.5.1
Figure 4.11 Embedding layer designed to take in a vocabulary of five unique items. You have to write the code that maps objects (like strings) into integers. The embedding maps each integer to its own d-dimensional vector x ∈ ℝd.
Visualizes the required separation between object-to-integer mapping and integer-to-vector embedding. section 4.2.2
