liveBook · Manning

Out-of-vocabulary word

As explained in LLMs in Production

An unknown token is a reserved token representing a word or text unit that a tokenizer or model does not recognize.

An out-of-vocabulary word is represented by a reserved <UNK> token when the tokenizer does not recognize it. Frequent <UNK> tokens reduce performance because they erase information from the input. section 2.1.1

The problem becomes concrete in code. A whitespace-based tokenizer can turn `def add_two_numbers_together(x, y):` into `[def, [UNK], y]`, losing punctuation and the function’s descriptive name. section 2.1.1

Treat unknown tokens as a tokenization-design warning:

  • Increase or adapt the vocabulary when ordinary words are repeatedly mapped to <UNK>. section 2.1.1
  • Choose a strategy suited to code, mathematics, multilingual text, and languages without alphabetic whitespace conventions. section 2.5 section 4.4.1
  • Prefer tokenization that preserves meaningful digits and text units instead of collapsing them into unknown tokens. section 4.4.1

Code completion

A tokenizer that recognizes only whitespace-separated words may fail on identifiers, punctuation, and syntax. A morphology-aware or subword-oriented strategy can preserve more of `add_two_numbers_together` for the model. section 2.1.1 section 4.4.1

Figure 4.3 Llama’s tokenization of the first arithmetic problem in the comparison table. Notice that each digit is highlighted individually, meaning that the model will eventually see all the digits.
figure

Use the digit-level tokenization figure to see how a strategy can expose every digit instead of hiding important numeric structure. section 4.4.1

How LLMs in Production covers it

How deep it goes
Explains the token’s role, performance consequences, and a worked code-completion failure, while deferring detailed tokenizer algorithms.
Examples
Uses a Python-like function identifier, digit-level tokenization, and arithmetic-tokenization comparisons to illustrate information loss and preservation.
What it assumes
Assumes basic familiarity with language models and tokens, while defining tokenization, vocabulary lookup, and unknown-token behavior.
What you can do after
Evaluate tokenization choices by checking whether code, numbers, and specialized text remain informative rather than becoming <UNK> tokens.
How it connects
Connects unknown tokens to tokenization, vocabulary, morphology, subword tokenization, code completion, multilingual processing, and language modeling.
Cover of LLMs in Production
liveBookmanning.com
Read LLMs in Production on liveBook Get the book at manning.com

Also explained in

Compare how 8 books explain out-of-vocabulary word →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLMs in Production. The section links point to the parts of the book it draws on.