liveBook · Manning

Vocabulary

As explained in Understanding AI

A vocabulary is the finalized collection of tokens that a language model can represent and process.

A vocabulary is the finalized collection of tokens a language model can represent. Before training, an automated tokenization process builds this collection from text, then assigns every token an arbitrary numeric ID. section 2.1.1

Byte Pair Encoding (BPE) builds a vocabulary in simple stages:

  • Start with individual characters as tokens. section 2.1.1
  • Find neighboring token pairs that occur most often and merge them. section 2.1.1
  • Repeat the merges until the vocabulary reaches its target size, typically 50,000 to 100,000 tokens. section 2.1.1
  • Assign each finalized token a numeric index; the number identifies its position but carries no meaning by itself. section 2.1.1

One word, several tokens

A common word such as “the” may become one token, while “unbelievable” may split into “un”, “believ”, and “able.” Humans see one word; the model receives three tokens and three numbers. section 2.1.1

Figure 2.1 The same word, seen in two ways. A human reads "unbelievable" as a single unit of meaning. A model never sees the word at all: it receives three tokens and three numbers.

Compare the human view of “unbelievable” with the model’s token-and-number representation. section 2.1.1

How Understanding AI covers it

How deep it goes
Traces BPE from character tokens through frequent-pair merges and arbitrary IDs, stopping before embeddings.
Examples
Uses “the,” “is,” and “unbelievable” to show single-token and subword representations.
What you can do after
Understand how token vocabularies are built and why token IDs do not encode meaning.
How it connects
Connects vocabulary with tokenization, Byte Pair Encoding, token IDs, and embeddings.

Also explained in

Compare how 17 books explain vocabulary →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Understanding AI. The section links point to the parts of the book it draws on.