Tokenization converts text into smaller units a language model can process. A token may be a word, word part, punctuation mark, space-like marker, or symbol; each token maps to an integer ID in the model’s vocabulary. section 10.2
A simple sentence
“Take aspirin today” might split into “Take”, “aspirin”, and “today”, though a real tokenizer may divide a word into smaller pieces. section 10.2



































