Language models do not receive text as people read it. Tokenization splits text into reusable pieces called tokens: whole words, word parts, punctuation, numbers, or spaces. Each token then receives a numerical ID, a label the computer can process rather than a definition or meaning. section 2.1.1
Tokenization handles language that a fixed word dictionary would struggle with:
- Short, common words may remain whole, while longer or less frequent words may split into smaller pieces. section 2.1.1
- Reusable pieces help process new words, names, mixed languages, typos, and variations. section 2.1.1
A word can become several tokens
A person may read “unbelievable” as one meaningful word, while a model receives three tokens and three numerical IDs. The sentence “The performance was unbelievable.” may therefore contain a different token pattern than its visible words suggest. section 2.1.1



































