liveBook · Manning

Tokenization

As explained in Machines that Think

A procedure for dividing text into discrete units for computational processing.

Tokenization divides text into discrete units called tokens. The boundaries chosen determine the resolution at which a model receives language: distinctions may be represented directly, reconstructed from smaller units, or lost through normalization. Splitting on spaces and punctuation is intuitive, but contractions, inflections, capitalization, and different writing systems complicate the choice. section 2.4.1 section 2.4.2

From words to fragments

A simple word-based split of “The quick brown fox jumps over the lazy dog.” produces nine word tokens and a punctuation token. Subword tokenization can instead divide an unfamiliar word such as “playfulness” into “play,” “ful,” and “ness,” using pieces encountered elsewhere. section 2.4.1 section 2.4.2

Tokenization schemes differ in the size of their units:

  • Word-level schemes are straightforward, but inflections, names, and rare terms can produce very large vocabularies. section 2.4.2
  • Character-level schemes need a much smaller vocabulary, but create longer sequences and leave the model to learn how characters combine into words. section 2.4.2
  • Subword schemes use reusable fragments to balance these extremes, allowing rare or new words to be represented as combinations of known pieces. section 2.4.2

Learned tokenizers use statistical patterns to choose units. Byte Pair Encoding (BPE) merges frequent pairs, which can yield fragments that reflect repetition rather than meaningful structure. WordPiece considers how merges affect segmentation across the corpus. SentencePiece works directly on raw text, treating whitespace as a symbol rather than assuming it marks word boundaries. section 2.4.4

Token boundaries also affect computational cost: each token takes space in a limited context window. Finer segmentation can preserve distinctions but lengthen sequences; coarser segmentation can be more compact while blurring detail. A scheme that is efficient for one writing system may fragment another. section 2.4.6

Tokens identify units, but do not by themselves express similarity or relationships between them; embeddings provide a representation for those connections. Splitting an unfamiliar term into known fragments also does not guarantee that its meaning is well learned, particularly when the term is rarely observed. section 2.5 section 2.6.4

How Machines that Think covers it

How deep it goes
Explains segmentation trade-offs and contrasts learned tokenizer approaches without providing implementation instructions.
Examples
Uses a sentence split into words and punctuation and divides “playfulness” into reusable subwords.
What you can do after
Evaluate tokenization choices by weighing linguistic coverage, vocabulary size, sequence length, and preserved distinctions.
How it connects
Connects tokenization to context windows, vocabulary size, and embeddings.

Also explained in

Compare how 36 books explain tokenization →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Machines that Think. The section links point to the parts of the book it draws on.