liveBook · Manning

Token id

As explained in Timeless Algorithms, The Foundational Ideas

An integer index that identifies a token in a model’s vocabulary.

Tokenization divides text into vocabulary units—such as words, word parts, or punctuation—and maps them to integer identifiers. The same tokenizer maps the same text to the same identifiers. An identifier is only an index: it does not encode the token’s meaning. Instead, it selects a row from a learned embedding matrix, providing the token’s initial numerical representation. section 12.2.1

Training adjusts the selected embedding along with other model parameters through backpropagation. The resulting representations capture statistical relationships in the contexts where tokens appear, rather than storing dictionary definitions. section 12.2.1

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Defines IDs as vocabulary indices, distinguishes them from meaning, and traces embedding lookup and subsequent parameter adjustment.
Examples
Uses illustrative embeddings for “trophy,” “box,” and “it” in attention calculations.
What it assumes
Connects token embeddings to neural-network forward passes and backpropagation rather than developing those ideas from scratch.
What you can do after
Understand that a token ID is an index, while its embedding supplies a learned numerical representation.
How it connects
Links token IDs to tokenization, vocabularies, embeddings, transformers, backpropagation, and prediction scores.

Also explained in

Compare how 10 books explain token id →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.