A token is a discrete unit of text used by a language model. Tokenization divides raw text into units the model can manipulate; depending on the scheme, a token may be a word, a fragment of a word, or a character. The boundaries are a modeling choice, not an automatic reflection of linguistic structure. section 2.4 section 2.4.1 section 2.4.2
Word boundaries are not equally obvious across languages. Spaces separate many English words, but Chinese and Japanese do not use them in the same way; inflections and compounds also complicate a word-based vocabulary. Choices about punctuation, capitalization, and normalization can preserve or discard distinctions that affect meaning. section 2.4.1 section 2.4.2
Tokenization schemes trade vocabulary size against sequence length:
- Word-level tokens are intuitive, but inflections, names, and rare terms can make the vocabulary very large. section 2.4.2
- Character-level tokens use a much smaller vocabulary, but produce longer sequences and require the model to learn how characters combine into words. section 2.4.2
- Subword tokens split text into reusable fragments. Common pieces can represent familiar patterns while combining to handle rare or new words. section 2.4.2
Building words from fragments
A subword tokenizer might divide “playfulness” into “play,” “ful,” and “ness,” allowing a model to process the word using pieces encountered elsewhere. Byte-pair encoding learns such units by repeatedly merging frequent adjacent symbols until a desired vocabulary size is reached. For example, it can represent “lowest” as “low” and “est.” section 2.4.2 section 2.4.3
Token boundaries determine the resolution at which text reaches the model. Finer divisions can preserve distinctions but lengthen sequences; coarser divisions can be more compact but blur detail. Each token also takes room in a limited context window, so segmentation affects how much text a model can process at once. section 2.4.6 section 2.4.7
A token begins as a discrete symbol: its identity alone does not reveal which other tokens are similar or related. Numerical representations can express relationships that tokenization by itself leaves unspecified. During generation, a model transforms representations of the active tokens using learned parameters to make some continuations more likely than others. section 2.4.7 section 2.5 section 6.4.1
A representation-sensitive task
Counting the letter “r” in “strawberry” can be a stress test: a model may process the word as whole-word or fragment tokens rather than as individual characters. Some newer models overcome such tasks through character-level steps or code, but the example illustrates why fluent language behavior need not imply equally reliable character-level manipulation. section 9.4.4







































