liveBook · Manning

Token id

As explained in Understanding AI

A token ID is a numerical identifier assigned to a token in a model's vocabulary.

When an LLM receives text, it does not receive words as people read them. It receives tokens: pieces of words, punctuation, numbers, or spaces. Each token receives a numerical identifier called a token ID. The ID is only a label, like a cloakroom ticket; it does not contain the token's meaning. section 2.1.1

The basic path from text to a model-ready representation is:

  • A tokenizer splits text into reusable pieces rather than relying on a fixed dictionary of whole words. section 2.1.1
  • The system maps each token to a numerical ID that computers can process. section 2.1.1
  • The model transforms each ID into an embedding, a vector whose learned relationships reflect how tokens appear in context. section 2.1.2

A word becomes several IDs

The word “unbelievable” may be split into “un”, “believ”, and “able”. These pieces can receive IDs such as 373, 9880, and 45794. Those numbers do not mean negation, trust, or ability. The later embedding step gives the model richer representations based on usage patterns. section 2.1.2

Figure 2.1 The same word, seen in two ways. A human reads "unbelievable" as a single unit of meaning. A model never sees the word at all: it receives three tokens and three numbers.

Compare the human view of “unbelievable” as one meaningful word with the model's view as three tokens and three numerical IDs. section 2.1.1

How Understanding AI covers it

How deep it goes
Defines IDs as meaningless labels and traces their conversion into embeddings, but does not detail tokenizer or embedding algorithms.
Examples
Uses “unbelievable” split into “un”, “believ”, and “able”, with example IDs 373, 9880, and 45794.
What you can do after
Apply it via OpenAI's online tokenizer to inspect token splits and token counts.
How it connects
Connects token IDs to tokenization, embeddings, attention, context windows, and next-token prediction.

Also explained in

Compare how 10 books explain token id →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Understanding AI. The section links point to the parts of the book it draws on.