liveBook · Manning

Token

As explained in LLM Evaluation and Alignment, The Foundational Ideas

A token is a basic unit of text processed by a language system, such as a word, punctuation mark, or subword piece.

A token is a basic unit of text processed by a system. In BLEU’s original formulation, tokens are typically words separated by whitespace and punctuation. Modern LLMs may split rare words into subword pieces; for BLEU and ROUGE, treating tokens as words works well. section 2.1.1

Word and subword units

The word “unhappiness” might be split into subword tokens such as “un”, “h”, “happiness”, and “ness.” section 2.1.1

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Distinguishes word-based BLEU tokens from modern subword pieces but does not detail tokenization algorithms.
Examples
Illustrates subword splitting with “unhappiness” and consecutive tokens with bigrams such as “the cat.”
What you can do after
Understand how text units become word-overlap counts in BLEU-style evaluation.
How it connects
Connects tokens to BLEU, ROUGE, and n-grams, where unigrams contain one token and bigrams contain two consecutive tokens.

Also explained in

Compare how 40 books explain token →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.