A tokenizer segments text into units that a model can process. Learned tokenizers make those boundaries part of the model’s representation, rather than relying only on fixed linguistic rules or preprocessing. section 2.4.4
Different methods prioritize different patterns. WordPiece favors merges that support consistent linguistic structure, even when they are not the most frequent character pairs. SentencePiece learns from raw text and treats whitespace as a symbol, so it does not depend on spaces to mark word boundaries. section 2.4.4 section 2.4.4
Token boundaries set the resolution at which text reaches a model. Smaller units can preserve nuance but lengthen sequences; larger units are more efficient but can lose detail. Since each token uses memory and context-window capacity, segmentation affects how much text fits and which distinctions remain available. section 2.4.6
Language and writing systems
SentencePiece can learn boundaries from scripts such as Japanese, Chinese, and Thai, where spaces may not mark word boundaries. Modern tokenizers can also adapt segmentation across languages such as English, Korean, and Arabic. section 2.4.4 section 2.4.4


















