A language-model vocabulary is the collection of unique tokens a tokenizer can represent, with each token assigned a numeric identifier. Tokens may be full words, word pieces, or smaller character sequences rather than words themselves. The model interacts with these identifiers, not directly with human-readable text. section 2.2.1 section 2.6
A tokenizer builds the vocabulary by processing training data: it normalizes text, segments it into tokens, and records each new token with a unique identifier. Byte-pair encoding (BPE) commonly learns frequent subword combinations instead of relying entirely on hand-written splitting rules. section 2.2.1 section 2.2.3
Vocabulary design balances representational coverage against practical cost:
- Larger vocabularies can represent more words and patterns directly, but increase model size, memory use, storage requirements, and computation. section 2.2.1 section 2.2.2
- Smaller vocabularies reduce those costs by reusing subwords, but may represent some content less effectively. section 2.2.2 section 2.6
- Out-of-vocabulary problems occur when text cannot be represented by any available token or combination of subwords. section 2.2.2
- Numeric token identifiers do not inherently express relationships such as shared prefixes, suffixes, spelling, or meaning; those relationships must be learned from observed data. section 2.6 section 2.2
From text to vocabulary entries
A simple space-based tokenizer might split “hello world” into “hello” and “world,” but “hello, world” can produce separate “hello,” and “hello” tokens. BPE instead learns frequent pieces, so words such as “schoolhouse” may be represented as “school” and “house,” while “thoughtful” may become “thought” and “ful.” section 2.2.3 section 2.2
Figure 2.2 Generically, tokenization involves processing input to produce numeric identifiers for tokens.
Trace the general pipeline from input text through tokenization to numeric identifiers. section 2.2.1
Figure 2.5 A simplified byte pair encoding algorithm for creating tokens: first, find the most frequent pair of characters “ng.” Next, replace all instances of “ng” with a placeholder token “T,” and add “ng” to the vocabulary. Repeat the process until no common byte pairs remain.
Follow a simplified BPE cycle: find a frequent character pair, merge it into a token, add it to the vocabulary, and repeat. section 2.2.3
