A vocabulary maps each unique text token—words, punctuation, or special markers—to a unique integer identifier. Token IDs provide the intermediate representation before embedding vectors and let a model encode input text and decode generated numbers back into text. section 2.3
Build and use a vocabulary through this data flow:
- Tokenize the training text, remove duplicates, sort the tokens, and enumerate them to establish the vocabulary size and token-to-ID mapping. section 2.3
- Maintain an inverse mapping from integer IDs to tokens so generated IDs can be decoded into text. section 2.3
- Add <|unk|> for tokens absent from the vocabulary and <|endoftext|> to separate unrelated documents. section 2.4
Python vocabulary and token IDs
A vocabulary built from the sample text contains 1,130 entries. Adding <|endoftext|> and <|unk|> increases it to 1,132. SimpleTokenizerV2 encodes "Hello, do you like tea?" by assigning 1131 to the unknown word Hello, and decodes that ID as <|unk|>. section 2.3 section 2.4
Figure 2.6 We build a vocabulary by tokenizing the entire text in a training dataset into individual tokens. These individual tokens are then sorted alphabetically, and duplicate tokens are removed. The unique tokens are then aggregated into a vocabulary that defines a mapping from each unique token to a unique integer value. The depicted vocabulary is purposely small and contains no punctuation or special characters for simplicity.
Shows how tokenized training data becomes a sorted set of unique tokens and then a token-to-integer vocabulary. section 2.3
