A tokenizer converts text into token IDs for model input and converts predicted IDs back into readable text. It uses a vocabulary that maps unique tokens to integers in both directions. section 2.3 section 5.1.1
A practical tokenizer follows this data flow:
- Build a vocabulary by tokenizing training text, removing duplicates, sorting tokens, and assigning integer IDs. section 2.3
- Encode text by splitting it into words and punctuation, then mapping each token through the vocabulary. section 2.3
- Decode IDs by reversing the mapping, joining tokens, and removing spacing inserted before punctuation. section 2.3
- Handle unseen words with <|unk|> and separate unrelated documents with <|endoftext|>. section 2.4
Encoding through GPT
The GPT pipeline encodes input text into token IDs, processes those IDs into logits, converts selected logits into new IDs, and decodes the result into text. A GPT-2 tokenizer and utility functions place the IDs in a batch-shaped tensor before generation. section 5.1.1
