Tokenization divides text into discrete units called tokens. The boundaries chosen determine the resolution at which a model receives language: distinctions may be represented directly, reconstructed from smaller units, or lost through normalization. Splitting on spaces and punctuation is intuitive, but contractions, inflections, capitalization, and different writing systems complicate the choice. section 2.4.1 section 2.4.2
From words to fragments
A simple word-based split of “The quick brown fox jumps over the lazy dog.” produces nine word tokens and a punctuation token. Subword tokenization can instead divide an unfamiliar word such as “playfulness” into “play,” “ful,” and “ness,” using pieces encountered elsewhere. section 2.4.1 section 2.4.2
Tokenization schemes differ in the size of their units:
- Word-level schemes are straightforward, but inflections, names, and rare terms can produce very large vocabularies. section 2.4.2
- Character-level schemes need a much smaller vocabulary, but create longer sequences and leave the model to learn how characters combine into words. section 2.4.2
- Subword schemes use reusable fragments to balance these extremes, allowing rare or new words to be represented as combinations of known pieces. section 2.4.2
Learned tokenizers use statistical patterns to choose units. Byte Pair Encoding (BPE) merges frequent pairs, which can yield fragments that reflect repetition rather than meaningful structure. WordPiece considers how merges affect segmentation across the corpus. SentencePiece works directly on raw text, treating whitespace as a symbol rather than assuming it marks word boundaries. section 2.4.4
Token boundaries also affect computational cost: each token takes space in a limited context window. Finer segmentation can preserve distinctions but lengthen sequences; coarser segmentation can be more compact while blurring detail. A scheme that is efficient for one writing system may fragment another. section 2.4.6
Tokens identify units, but do not by themselves express similarity or relationships between them; embeddings provide a representation for those connections. Splitting an unfamiliar term into known fragments also does not guarantee that its meaning is well learned, particularly when the term is rarely observed. section 2.5 section 2.6.4



































