Tokenization turns a prompt such as “a tall tree in the forest” into smaller parts that can be assigned numbers and processed by a neural network. Machines need this translation because machine learning ultimately operates on numbers. section 4.3.2
Two straightforward ways to split text illustrate the trade-offs:
- Character tokenization assigns numbers to each character, including spaces. It is simple and can work surprisingly well, but individual characters carry little meaning. section 4.3.2
- Word tokenization assigns numbers from a large vocabulary. Words preserve more meaning, but representing every possible English word requires tens or hundreds of thousands of numbers. section 4.3.2
A tiny prompt experiment
Take “All mimsy were the borogoves” from Jabberwocky. A tokenizer might represent it character by character or map its words through a vocabulary; the resulting number sequence feeds the model. section 4.3.2



































