liveBook · Manning

Tokenizer

As explained in Machines that Think

A tokenizer is a software component that segments text into discrete units for computational language processing.

A tokenizer segments text into units that a model can process. Learned tokenizers make those boundaries part of the model’s representation, rather than relying only on fixed linguistic rules or preprocessing. section 2.4.4

Different methods prioritize different patterns. WordPiece favors merges that support consistent linguistic structure, even when they are not the most frequent character pairs. SentencePiece learns from raw text and treats whitespace as a symbol, so it does not depend on spaces to mark word boundaries. section 2.4.4 section 2.4.4

Token boundaries set the resolution at which text reaches a model. Smaller units can preserve nuance but lengthen sequences; larger units are more efficient but can lose detail. Since each token uses memory and context-window capacity, segmentation affects how much text fits and which distinctions remain available. section 2.4.6

Language and writing systems

SentencePiece can learn boundaries from scripts such as Japanese, Chinese, and Thai, where spaces may not mark word boundaries. Modern tokenizers can also adapt segmentation across languages such as English, Korean, and Arabic. section 2.4.4 section 2.4.4

How Machines that Think covers it

How deep it goes
Compares learned segmentation approaches and explains how token granularity affects nuance, sequence length, memory, and context capacity.
Examples
Contrasts whitespace-free Japanese, Chinese, and Thai with multilingual processing that includes English, Korean, and Arabic.
What you can do after
Evaluate how segmentation choices trade preserved detail against sequence length and computational efficiency.
How it connects
Connects tokenization with WordPiece, SentencePiece, multilingual modeling, and context windows.

Also explained in

Compare how 19 books explain tokenizer →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Machines that Think. The section links point to the parts of the book it draws on.