liveBook · Manning

Tokenization

As explained in How Large Language Models Work

Tokenization is the process of converting an input into discrete units or a sequence of numeric representations for model processing.

Tokenization converts text into discrete pieces that an LLM can process as numbers. A token may be a whole word, part of a word, or a subword; the model works with numeric identifiers rather than directly seeing the characters or their meanings. section 2.2 section 2.6

The pipeline receives a string, normalizes it when appropriate, segments it into tokens, and maps each token to a unique integer. Normalization might change case, remove punctuation, or standardize Unicode; segmentation determines how text is split. section 2.2.1 section 2.2.2

Figure 2.2 Generically, tokenization involves processing input to produce numeric identifiers for tokens.
figure

Use the pipeline diagram to follow text from input through transformation and segmentation to numeric token identifiers. section 2.2.1

Several design choices make tokenization useful but imperfect:

  • Byte-pair encoding (BPE) finds frequent character pairs and combines them into reusable subword tokens, reducing the need for hand-written splitting rules. section 2.2.3
  • Vocabulary size balances coverage against storage and computation: a small vocabulary may fail to represent unfamiliar text, while a large one increases model resources. section 2.2.1 section 2.2.2
  • Token counts do not reliably measure character or sentence length because frequent strings may receive one token while rare variants are split into several. section 2.2.4
  • Representation affects capability: inconsistent digit tokens make arithmetic harder, and languages with less training data may require more tokens and cost more to process. section 2.3.2 section 2.3.3

Why the split matters

A whitespace tokenizer can turn “hello, world” into [hello,, world], treating “hello” and “hello,” as different tokens. BPE instead learns frequent subword combinations. Numbers expose a sharper problem: a sequence such as 123456 may be grouped into chunks rather than individual digits, making positional arithmetic harder; digit-level tokenization can improve arithmetic learning. section 2.2.3 section 2.3.2

Tokenization is therefore part of an LLM’s practical boundary, not merely an invisible preprocessing step. It influences what text can be represented efficiently, how much it costs to process, and how well a model handles code, mathematics, languages, or unusual characters. When an application struggles, the tokenizer may be part of the problem, although fixing it can require substantial engineering. section 2.3 section 2.5

How How Large Language Models Work covers it

How deep it goes
Explains normalization, segmentation, numeric mapping, vocabulary trade-offs, BPE, and capability risks without dense implementation detail.
Examples
Uses whitespace splitting, BPE pair merging, running versus runnin, digit-level arithmetic, multilingual token costs, and code examples.
What it assumes
Assumes basic text-processing familiarity while defining tokens, vocabulary, normalization, segmentation, and BPE from the ground up.
What you can do after
Evaluate tokenizer effects on vocabulary, cost, language coverage, arithmetic, code, and application-specific representation.
How it connects
Connects tokenization with feature engineering, normalization, segmentation, vocabulary, byte-pair encoding, arithmetic, and language equity.
Cover of How Large Language Models Work
liveBookmanning.com
Read How Large Language Models Work on liveBook Get the book at manning.com

Also explained in

Compare how 25 books explain tokenization →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of How Large Language Models Work. The section links point to the parts of the book it draws on.