liveBook · Manning

Out-of-vocabulary word

Also called unknown token, [UNK]

An out-of-vocabulary word or text unit is absent from a tokenizer’s or model’s recognized vocabulary and may be represented by a special unknown-token marker.

How 8 Manning books explain it

Natural Language Processing in Action, Second Edition

An out-of-vocabulary token is a token that is absent from the vocabulary or pretrained representation available to a model.

How it is used
Motivates vocabulary and tokenization choices for handling words absent from a model’s vocabulary.
How deep it goes
Explains shared unknown-token counting and character fallback, with a corpus-level count of unknown tokens.
Examples
Uses a masked WikiText2 passage and counts its training-set unknown tokens.
What you can do after
Evaluate how unknown-token aggregation or character fallback affects a pipeline’s handling of unseen words.

Read how Natural Language Processing in Action, Second Edition explains out-of-vocabulary word →

LLMs in Production

An unknown token is a reserved token representing a word or text unit that a tokenizer or model does not recognize.

How it is used
Diagnoses information loss caused by tokenizers that map unfamiliar words, identifiers, numbers, or syntax to <UNK>.
How deep it goes
Explains the token’s role, performance consequences, and a worked code-completion failure, while deferring detailed tokenizer algorithms.
Examples
Uses a Python-like function identifier, digit-level tokenization, and arithmetic-tokenization comparisons to illustrate information loss and preservation.
What you can do after
Evaluate tokenization choices by checking whether code, numbers, and specialized text remain informative rather than becoming <UNK> tokens.

Read how LLMs in Production explains out-of-vocabulary word →

Learn Generative AI with PyTorch

How it is used
Prevents missing vocabulary entries from stopping text encoding and can provide a prompt for unconditional generation.
How deep it goes
Shows vocabulary construction, unknown-token indexing, and generation use, without detailing the model’s internal handling of UNK.
Examples
Uses “technology” as a missing word and UNK as a generation prompt for Hemingway-style text.
What you can do after
Apply an UNK fallback when converting tokens to integer IDs, or use UNK to prompt unconditional generation.

Read how Learn Generative AI with PyTorch explains out-of-vocabulary word →

Deep Learning with R, Third Edition

An unknown token is a reserved token representing text units that are absent from a tokenizer's retained vocabulary.

How it is used
Handles text terms missing from a limited vocabulary during tokenization and detokenization.
How deep it goes
Explains fallback indices and detokenization implementation, but does not develop subword-tokenization alternatives.
Examples
Uses a small dog-sentence vocabulary and a 2,000-term Moby Dick vocabulary to demonstrate unknown-word indices and "[UNK]" output.
What you can do after
Build it from scratch by mapping unmatched R tokenizer terms to index 0 and rendering them as "[UNK]".

Read how Deep Learning with R, Third Edition explains out-of-vocabulary word →

How Large Language Models Work

An unknown token is a reserved token emitted when a tokenizer cannot process an input symbol correctly.

How it is used
Explains how tokenizers handle symbols that vocabulary and subword combinations cannot represent.
How deep it goes
Defines the failure, identifies [UNK], and relates it to vocabulary size, normalization, and information loss without detailing implementation internals.
Examples
Uses an unrepresentable word, the [UNK] marker, and mixed-case variants such as “LLMs,” “LLms,” and “llms.”
What you can do after
Evaluate how vocabulary and normalization choices affect tokenizer coverage, model capability, accuracy, and information loss.

Read how How Large Language Models Work explains out-of-vocabulary word →

Build a Large Language Model (From Scratch)

A placeholder token used by some tokenizers for out-of-vocabulary words, which BPE avoids here by subword decomposition.

How it is used
Marks words absent from a tokenizer vocabulary so encoding can proceed.
How deep it goes
Explains vocabulary insertion, integer mapping, encoding, decoding, and BPE’s alternative, but defers detailed BPE implementation.
Examples
Demonstrates <|unk|> with unseen words such as “Hello,” “palace,” and “someunknownPlace.”
What you can do after
Apply via SimpleTokenizerV2 by adding <|unk|>, encoding unseen words, and decoding their placeholder IDs.

Read how Build a Large Language Model (From Scratch) explains out-of-vocabulary word →

Build a Text-to-Image Generator (from Scratch)

An unknown token is a special marker representing a token not present in the model's vocabulary.

How it is used
Converts tokens absent from a vocabulary into reserved indices so prompts and captions can enter transformer pipelines.
How deep it goes
Traces dictionary construction, fallback lookup, and reverse mapping, stopping at token representation rather than model behavior.
Examples
Uses English and German translation dictionaries plus an image-captioning vocabulary with <unk> assigned index 3.
What you can do after
Builds token-index dictionaries that safely encode missing vocabulary entries for transformer inputs.

Read how Build a Text-to-Image Generator (from Scratch) explains out-of-vocabulary word →

Deep Learning with Python, Third Edition

A special unknown-token marker used when a tokenizer cannot represent a word with known vocabulary items.

How it is used
Uses an unknown-token marker to index input terms absent from a text-processing vocabulary.
How deep it goes
Explains vocabulary lookup and fallback indexing with a worked sentence, but does not discuss subword tokenization.
Examples
Uses a fox-and-dog sentence, a Python vocabulary dictionary, the word “lazy,” and the resulting index sequence.
What you can do after
Apply it via a vocabulary lookup that maps missing tokens to a reserved [UNK] index.

Read how Deep Learning with Python, Third Edition explains out-of-vocabulary word →

Also covered in

What to know first

Related concepts

Questions about out-of-vocabulary word

What is out-of-vocabulary word?

An out-of-vocabulary word or text unit is absent from a tokenizer’s or model’s recognized vocabulary and may be represented by a special unknown-token marker.

Which Manning books explain out-of-vocabulary word?

Natural Language Processing in Action, Second Edition, LLMs in Production, Learn Generative AI with PyTorch, Deep Learning with R, Third Edition, How Large Language Models Work, Build a Large Language Model (From Scratch), Build a Text-to-Image Generator (from Scratch) and Deep Learning with Python, Third Edition.

What should I know before learning out-of-vocabulary word?

Manning books that teach out-of-vocabulary word usually introduce language model, token, Unicode and accuracy first.

What is the difference between out-of-vocabulary word and prompt?

An out-of-vocabulary word or text unit is absent from a tokenizer’s or model’s recognized vocabulary and may be represented by a special unknown-token marker. A prompt is text or other input supplied to an AI model to provide instructions, context, or other information that guides its response or generated output.

What is out-of-vocabulary word a kind of?

Out-of-vocabulary word is a kind of token. A token is a discrete unit of text, such as a word, subword, punctuation mark, or symbol, produced by tokenization and processed by a language model.

What is out-of-vocabulary word also called?

Out-of-vocabulary word is also called unknown token or [UNK].

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.