liveBook · Manning

Token

As explained in Machines that Think

A discrete unit of text used in computational language processing, such as a character, word fragment, or word.

A token is a discrete unit of text used by a language model. Tokenization divides raw text into units the model can manipulate; depending on the scheme, a token may be a word, a fragment of a word, or a character. The boundaries are a modeling choice, not an automatic reflection of linguistic structure. section 2.4 section 2.4.1 section 2.4.2

Word boundaries are not equally obvious across languages. Spaces separate many English words, but Chinese and Japanese do not use them in the same way; inflections and compounds also complicate a word-based vocabulary. Choices about punctuation, capitalization, and normalization can preserve or discard distinctions that affect meaning. section 2.4.1 section 2.4.2

Tokenization schemes trade vocabulary size against sequence length:

  • Word-level tokens are intuitive, but inflections, names, and rare terms can make the vocabulary very large. section 2.4.2
  • Character-level tokens use a much smaller vocabulary, but produce longer sequences and require the model to learn how characters combine into words. section 2.4.2
  • Subword tokens split text into reusable fragments. Common pieces can represent familiar patterns while combining to handle rare or new words. section 2.4.2

Building words from fragments

A subword tokenizer might divide “playfulness” into “play,” “ful,” and “ness,” allowing a model to process the word using pieces encountered elsewhere. Byte-pair encoding learns such units by repeatedly merging frequent adjacent symbols until a desired vocabulary size is reached. For example, it can represent “lowest” as “low” and “est.” section 2.4.2 section 2.4.3

Token boundaries determine the resolution at which text reaches the model. Finer divisions can preserve distinctions but lengthen sequences; coarser divisions can be more compact but blur detail. Each token also takes room in a limited context window, so segmentation affects how much text a model can process at once. section 2.4.6 section 2.4.7

A token begins as a discrete symbol: its identity alone does not reveal which other tokens are similar or related. Numerical representations can express relationships that tokenization by itself leaves unspecified. During generation, a model transforms representations of the active tokens using learned parameters to make some continuations more likely than others. section 2.4.7 section 2.5 section 6.4.1

A representation-sensitive task

Counting the letter “r” in “strawberry” can be a stress test: a model may process the word as whole-word or fragment tokens rather than as individual characters. Some newer models overcome such tasks through character-level steps or code, but the example illustrates why fluent language behavior need not imply equally reliable character-level manipulation. section 9.4.4

How Machines that Think covers it

How deep it goes
Explains word, character, and subword trade-offs and outlines byte-pair encoding through repeated frequency-based merges.
Examples
Illustrates subword segmentation with “playfulness” and tests character-level brittleness with letter counting in “strawberry.”
What you can do after
Evaluate how token granularity affects vocabulary size, sequence length, and the distinctions available to a model.
How it connects
Connects tokenization to embeddings, context windows, and next-token prediction.

Also explained in

Compare how 40 books explain token →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Machines that Think. The section links point to the parts of the book it draws on.