Out-of-vocabulary word
An out-of-vocabulary word or text unit is absent from a tokenizer’s or model’s available vocabulary and may be represented by a special unknown-token marker.
Natural Language Processing in Action, Second Edition
An out-of-vocabulary token is a token that is absent from the vocabulary or pretrained representation available to a model.
- How it is used
- Handles tokens excluded from a word-based vocabulary while motivating choices about vocabulary size and tokenization.
- How deep it goes
- Defines the generic OOV token and explains its information loss, then examines a masked dataset; does not develop an OOV representation model.
- Examples
- Uses an unusual invented word grouped under <OOV> and Python counts of <unk> in WikiText2.
- What you can do after
- Diagnose/debug with it by checking whether unknown markers reflect excluded vocabulary items or dataset masking before interpreting model behavior.
Read how Natural Language Processing in Action, Second Edition explains out-of-vocabulary word →
LLMs in Production
An unknown token is a reserved token representing a word or text unit that a tokenizer or model does not recognize.
- How it is used
- Diagnoses information loss caused by tokenizers that map unfamiliar words, identifiers, numbers, or syntax to <UNK>.
- How deep it goes
- Explains the token’s role, performance consequences, and a worked code-completion failure, while deferring detailed tokenizer algorithms.
- Examples
- Uses a Python-like function identifier, digit-level tokenization, and arithmetic-tokenization comparisons to illustrate information loss and preservation.
- What you can do after
- Evaluate tokenization choices by checking whether code, numbers, and specialized text remain informative rather than becoming <UNK> tokens.
Read how LLMs in Production explains out-of-vocabulary word →
Learn Generative AI with PyTorch
- How it is used
- Prevents vocabulary lookup failures by assigning unseen tokens a known marker and index.
- How deep it goes
- Defines the UNK fallback and shows vocabulary indexing, but does not detail alternative unknown-token strategies.
- Examples
- Contrasts missing “technology” with French split into “fr” and “ench,” and uses UNK as a text-generation prompt.
- What you can do after
- Operate a token dictionary by mapping absent entries to UNK before converting tokens to indexes.
Read how Learn Generative AI with PyTorch explains out-of-vocabulary word →
Deep Learning with R, Third Edition
An unknown token is a reserved token representing text units that are absent from a tokenizer's retained vocabulary.
- How it is used
- Handles text terms missing from a limited vocabulary during tokenization and detokenization.
- How deep it goes
- Explains fallback indices and detokenization implementation, but does not develop subword-tokenization alternatives.
- Examples
- Uses a small dog-sentence vocabulary and a 2,000-term Moby Dick vocabulary to demonstrate unknown-word indices and "[UNK]" output.
- What you can do after
- Build it from scratch by mapping unmatched R tokenizer terms to index 0 and rendering them as "[UNK]".
Read how Deep Learning with R, Third Edition explains out-of-vocabulary word →
How Large Language Models Work
An unknown token is a reserved token emitted when a tokenizer cannot process an input symbol correctly.
- How it is used
- Explains how tokenizers handle symbols that vocabulary and subword combinations cannot represent.
- How deep it goes
- Defines the failure, identifies [UNK], and relates it to vocabulary size, normalization, and information loss without detailing implementation internals.
- Examples
- Uses an unrepresentable word, the [UNK] marker, and mixed-case variants such as “LLMs,” “LLms,” and “llms.”
- What you can do after
- Evaluate how vocabulary and normalization choices affect tokenizer coverage, model capability, accuracy, and information loss.
Read how How Large Language Models Work explains out-of-vocabulary word →
Build a Large Language Model (From Scratch)
A placeholder token used by some tokenizers for out-of-vocabulary words, which BPE avoids here by subword decomposition.
- How it is used
- Marks words absent from a tokenizer vocabulary so encoding can proceed.
- How deep it goes
- Explains vocabulary insertion, integer mapping, encoding, decoding, and BPE’s alternative, but defers detailed BPE implementation.
- Examples
- Demonstrates <|unk|> with unseen words such as “Hello,” “palace,” and “someunknownPlace.”
- What you can do after
- Apply via SimpleTokenizerV2 by adding <|unk|>, encoding unseen words, and decoding their placeholder IDs.
Read how Build a Large Language Model (From Scratch) explains out-of-vocabulary word →
Build a Text-to-Image Generator (from Scratch)
An unknown token is a special marker representing a token not present in the model's vocabulary.
- How it is used
- Converts tokens absent from a vocabulary into reserved indices so prompts and captions can enter transformer pipelines.
- How deep it goes
- Traces dictionary construction, fallback lookup, and reverse mapping, stopping at token representation rather than model behavior.
- Examples
- Uses English and German translation dictionaries plus an image-captioning vocabulary with <unk> assigned index 3.
- What you can do after
- Builds token-index dictionaries that safely encode missing vocabulary entries for transformer inputs.
Read how Build a Text-to-Image Generator (from Scratch) explains out-of-vocabulary word →
Deep Learning with Python, Third Edition
A special unknown-token marker used when a tokenizer cannot represent a word with known vocabulary items.
- How it is used
- Uses an unknown-token marker to index input terms absent from a text-processing vocabulary.
- How deep it goes
- Explains vocabulary lookup and fallback indexing with a worked sentence, but does not discuss subword tokenization.
- Examples
- Uses a fox-and-dog sentence, a Python vocabulary dictionary, the word “lazy,” and the resulting index sequence.
- What you can do after
- Apply it via a vocabulary lookup that maps missing tokens to a reserved [UNK] index.
Read how Deep Learning with Python, Third Edition explains out-of-vocabulary word →
Related concepts
- subword tokenizationaddressed by · Subword tokenization is a method of representing text with tokens that may be whole words or smaller word fragments, allowing unfamiliar words to be encoded as combinations of units.
- tokenkind of · A text token is a discrete unit produced by tokenization and processed as one element, such as a word, subword, character, symbol, or punctuation mark.
- tokenizerused by · A tokenizer is a text-processing component or algorithm that segments text into discrete tokens and may map them to numerical identifiers according to defined rules or a vocabulary.
- promptalternative to · A prompt is an instruction, question, or cue presented to a person or computational system to guide or elicit a response.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.