liveBook · Manning

Out-of-vocabulary word

As explained in How Large Language Models Work

An unknown token is a reserved token emitted when a tokenizer cannot process an input symbol correctly.

An out-of-vocabulary word is text that a tokenizer cannot represent using its available tokens, even by combining known subwords. The tokenizer may emit an unknown token, commonly written as [UNK]. section 2.2.2 section 2.2.3

This is a tokenizer failure, not evidence that the model understands a new word imperfectly. If the text cannot be represented, the model cannot capture information from that piece of text. Larger vocabularies reduce this risk but increase model size, so token design balances coverage against efficiency. section 2.2.2

A word outside the vocabulary

Suppose an unusual word is absent from the vocabulary and no available subword sequence can represent it. The tokenizer emits [UNK], replacing the original symbol with a generic marker and losing information about that text. section 2.2.2 section 2.2.3

Figure 2.3 The normalization process commonly involves changing text to remove uppercase characters and punctuation.
figure

Normalization can reduce out-of-vocabulary pressure by mapping variants such as “LLMs,” “LLms,” and “llms” to one simpler form, though it may also remove distinctions. section 2.2.2

How How Large Language Models Work covers it

How deep it goes
Defines the failure, identifies [UNK], and relates it to vocabulary size, normalization, and information loss without detailing implementation internals.
Examples
Uses an unrepresentable word, the [UNK] marker, and mixed-case variants such as “LLMs,” “LLms,” and “llms.”
What you can do after
Evaluate how vocabulary and normalization choices affect tokenizer coverage, model capability, accuracy, and information loss.
How it connects
Connects out-of-vocabulary words to tokenization, subwords, vocabulary size, normalization, byte pair encoding, WordPiece, and SentencePiece.
Cover of How Large Language Models Work
liveBookmanning.com
Read How Large Language Models Work on liveBook Get the book at manning.com

Also explained in

Compare how 8 books explain out-of-vocabulary word →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of How Large Language Models Work. The section links point to the parts of the book it draws on.