Subword tokenization

As explained in How Large Language Models Work

A subword is a word fragment that lies between a complete word and an individual character and can carry part of a word's meaning.

A subword is a word fragment between a complete word and a single character. It can carry part of a word’s meaning: schoolhouse may become school and house, while thoughtful may become thought and ful. section 2.2

Subwords help language models represent text through tokenization:

• Byte-pair encoding starts with individual characters, repeatedly combines frequent neighboring pairs, and stops when the vocabulary reaches a suitable size. section 2.2.3
• The resulting vocabulary balances compact representation against coverage: a vocabulary that is too small may fail to represent unfamiliar words. section 2.2.2
• Token identifiers do not inherently preserve relationships such as shared prefixes, suffixes, capitalization, or similar spelling; models learn such relationships from data. section 2.2 section 2.6

A frequent-pair merge

Imagine that the adjacent letters n and g frequently occur together. Byte-pair encoding can merge them into an ng token, replace those pairs, add ng to the vocabulary, and continue merging other frequent pairs. section 2.2.3

Figure 2.5 A simplified byte pair encoding algorithm for creating tokens: first, find the most frequent pair of characters “ng.” Next, replace all instances of “ng” with a placeholder token “T,” and add “ng” to the vocabulary. Repeat the process until no common byte pairs remain.

The figure traces byte-pair encoding from frequent character pairs to merged subword tokens. section 2.2.3

How How Large Language Models Work covers it

How deep it goes

Explains subword meaning, BPE construction, vocabulary tradeoffs, and token-relationship limitations, but stops short of model-training internals.

Examples

Uses schoolhouse, thoughtful, and repeated merging of the frequent letter pair ng to demonstrate subword segmentation.

What you can do after

Evaluate how tokenization represents unfamiliar words and why vocabulary choices affect model coverage, storage, and computation.

How it connects

Connects subwords with tokenization, byte-pair encoding, vocabularies, normalization, and out-of-vocabulary problems.

liveBookmanning.com

Read How Large Language Models Work on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Natural Language Processing in Action, Second EditionA subword is a word fragment, character sequence, or complete word treated as a unit in subword-based language modeling.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionA subword token is a token representing a complete word, a word fragment, or a character-level fallback unit.
• 

  liveBookmanning.com
  Learn Generative AI with PyTorchA partial word unit produced by subword tokenization, smaller than a whole word but larger than a character.
• 

  liveBookmanning.com
  Deep Learning with Python, Third EditionA tokenization method that splits text into units smaller than words but larger than characters to balance vocabulary size and sequence length.
• 

  liveBookmanning.com
  LLMs in ProductionA subword is a tokenization unit corresponding to part of a word rather than necessarily a complete word.
• 

  liveBookmanning.com
  Build a Large Language Model (From Scratch)A token fragment smaller than a full word, used by subword tokenizers to represent rare or unseen words.

Compare how 7 books explain subword tokenization →

Related concepts

• tokenizationkind of · Tokenization is the process of dividing text or other sequential input into discrete units called tokens for computational processing.
• byte pair encodingincludes · Byte-pair encoding is a subword tokenization method that iteratively merges frequent adjacent symbol pairs to build a vocabulary of larger units.
• SentencePieceused by · SentencePiece is a language-independent software library that learns and applies subword tokenization directly to raw text without requiring whitespace-based pretokenization.
• out-of-vocabulary wordaddresses · An out-of-vocabulary word or text unit is absent from a tokenizer’s or model’s available vocabulary and may be represented by a special unknown-token marker.
• FastTextused by · FastText is a word-representation model that composes word vectors from character n-grams and whole-word information, including subword representations.

This explainer was generated by Manning's knowledge graph from the text of How Large Language Models Work. The section links point to the parts of the book it draws on.