A token is a small, discrete text unit that a language model processes and predicts. Depending on the tokenizer, it can represent a full word, part of a word, or punctuation. Tokenized text is converted into numerical IDs that the model can ingest. section 1.2
Example token sequence
The sentence “An LLM can be useful.” may become “An”, “ L”, “LM”, “ can”, “ be”, “ useful”, and “.”. During pretraining, the model learns to predict the next token from large collections of such sequences. section 1.2
Tokens connect text processing to the standard training pipeline: massive unlabeled datasets are represented as token sequences, and pretraining optimizes next-token prediction. The tokenizer is separate from the language model but remains essential to text processing and generation. section 1.2
