liveBook · Manning

Tokenization

As explained in Timeless Algorithms, The Foundational Ideas

The process of dividing text into discrete units called tokens.

Before a transformer can process text, tokenization divides it into discrete units from a vocabulary. A token may be a whole word, part of a word, punctuation, or another recurring text element. The tokenizer converts the text into a sequence of integer identifiers; applying the same tokenizer to the same text produces the same identifiers. section 12.2.1

An identifier is an index, not a numerical description of meaning. It selects a row from a learned embedding matrix, giving the token its initial vector representation. Training adjusts embeddings along with other model parameters, allowing them to capture statistical relationships among token contexts. section 12.2.1

These representations enter the transformer’s attention calculations. In a simplified pronoun-resolution example, vectors for “trophy,” “box,” and “it” illustrate how attention combines contextual information; the values are illustrative, not outputs from a trained model. section 12.2.1

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Defines token types and identifiers, then explains their connection to learned embeddings without detailing a tokenization algorithm.
Examples
Uses an illustrative pronoun-resolution calculation involving tokens for “trophy,” “box,” and “it.”
What you can do after
Evaluate and understand how token identifiers differ from learned vector representations.
How it connects
Connects tokenization to vocabulary, embeddings, backpropagation, and transformer attention.

Also explained in

Compare how 36 books explain tokenization →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.