A token ID is an integer assigned by a vocabulary to a token. It converts tokenized text into discrete model input, which embedding layers later map to vectors. section 2.3 section 2.7
Token IDs move through the GPT-style input pipeline as follows:
- Build a vocabulary from unique tokens and assign each token a unique integer. section 2.3
- Encode new text by tokenizing it and replacing each token with its vocabulary ID; decode by reversing the mapping. section 2.3
- Feed ID sequences to an embedding layer, which retrieves one learned weight-matrix row for each ID. section 2.7
- Combine token embeddings with positional embeddings before passing them to the main language-model layers. section 2.8
- Use special IDs for unknown tokens, text boundaries, or padding when vocabulary coverage or batch lengths differ. section 2.4 section 6.3
Small embedding lookup
With vocabulary size 6 and embedding dimension 3, input IDs [2, 3, 5, 1] produce a tensor shaped [4, 3]. ID 3 retrieves the fourth row of the embedding matrix because indexing starts at zero. section 2.7
Figure 2.6 We build a vocabulary by tokenizing the entire text in a training dataset into individual tokens. These individual tokens are then sorted alphabetically, and duplicate tokens are removed. The unique tokens are then aggregated into a vocabulary that defines a mapping from each unique token to a unique integer value. The depicted vocabulary is purposely small and contains no punctuation or special characters for simplicity.
The vocabulary diagram shows how tokenization, duplicate removal, sorting, and integer assignment create the mapping that produces token IDs. section 2.3
