Out-of-vocabulary word
An out-of-vocabulary word or text unit is absent from a tokenizer’s or model’s recognized vocabulary and may be represented by a special unknown-token marker.
Natural Language Processing in Action, Second Edition
An out-of-vocabulary token is a token that is absent from the vocabulary or pretrained representation available to a model.
- How it is used
- Motivates vocabulary and tokenization choices for handling words absent from a model’s vocabulary.
- How deep it goes
- Explains shared unknown-token counting and character fallback, with a corpus-level count of unknown tokens.
- Examples
- Uses a masked WikiText2 passage and counts its training-set unknown tokens.
- What you can do after
- Evaluate how unknown-token aggregation or character fallback affects a pipeline’s handling of unseen words.
Read how Natural Language Processing in Action, Second Edition explains out-of-vocabulary word →
LLMs in Production
An unknown token is a reserved token representing a word or text unit that a tokenizer or model does not recognize.
- How it is used
- Diagnoses information loss caused by tokenizers that map unfamiliar words, identifiers, numbers, or syntax to <UNK>.
- How deep it goes
- Explains the token’s role, performance consequences, and a worked code-completion failure, while deferring detailed tokenizer algorithms.
- Examples
- Uses a Python-like function identifier, digit-level tokenization, and arithmetic-tokenization comparisons to illustrate information loss and preservation.
- What you can do after
- Evaluate tokenization choices by checking whether code, numbers, and specialized text remain informative rather than becoming <UNK> tokens.
Read how LLMs in Production explains out-of-vocabulary word →
Learn Generative AI with PyTorch
- How it is used
- Prevents missing vocabulary entries from stopping text encoding and can provide a prompt for unconditional generation.
- How deep it goes
- Shows vocabulary construction, unknown-token indexing, and generation use, without detailing the model’s internal handling of UNK.
- Examples
- Uses “technology” as a missing word and UNK as a generation prompt for Hemingway-style text.
- What you can do after
- Apply an UNK fallback when converting tokens to integer IDs, or use UNK to prompt unconditional generation.
Read how Learn Generative AI with PyTorch explains out-of-vocabulary word →
Deep Learning with R, Third Edition
An unknown token is a reserved token representing text units that are absent from a tokenizer's retained vocabulary.
- How it is used
- Handles text terms missing from a limited vocabulary during tokenization and detokenization.
- How deep it goes
- Explains fallback indices and detokenization implementation, but does not develop subword-tokenization alternatives.
- Examples
- Uses a small dog-sentence vocabulary and a 2,000-term Moby Dick vocabulary to demonstrate unknown-word indices and "[UNK]" output.
- What you can do after
- Build it from scratch by mapping unmatched R tokenizer terms to index 0 and rendering them as "[UNK]".
Read how Deep Learning with R, Third Edition explains out-of-vocabulary word →
How Large Language Models Work
An unknown token is a reserved token emitted when a tokenizer cannot process an input symbol correctly.
- How it is used
- Explains how tokenizers handle symbols that vocabulary and subword combinations cannot represent.
- How deep it goes
- Defines the failure, identifies [UNK], and relates it to vocabulary size, normalization, and information loss without detailing implementation internals.
- Examples
- Uses an unrepresentable word, the [UNK] marker, and mixed-case variants such as “LLMs,” “LLms,” and “llms.”
- What you can do after
- Evaluate how vocabulary and normalization choices affect tokenizer coverage, model capability, accuracy, and information loss.
Read how How Large Language Models Work explains out-of-vocabulary word →
Build a Large Language Model (From Scratch)
A placeholder token used by some tokenizers for out-of-vocabulary words, which BPE avoids here by subword decomposition.
- How it is used
- Marks words absent from a tokenizer vocabulary so encoding can proceed.
- How deep it goes
- Explains vocabulary insertion, integer mapping, encoding, decoding, and BPE’s alternative, but defers detailed BPE implementation.
- Examples
- Demonstrates <|unk|> with unseen words such as “Hello,” “palace,” and “someunknownPlace.”
- What you can do after
- Apply via SimpleTokenizerV2 by adding <|unk|>, encoding unseen words, and decoding their placeholder IDs.
Read how Build a Large Language Model (From Scratch) explains out-of-vocabulary word →
Build a Text-to-Image Generator (from Scratch)
An unknown token is a special marker representing a token not present in the model's vocabulary.
- How it is used
- Converts tokens absent from a vocabulary into reserved indices so prompts and captions can enter transformer pipelines.
- How deep it goes
- Traces dictionary construction, fallback lookup, and reverse mapping, stopping at token representation rather than model behavior.
- Examples
- Uses English and German translation dictionaries plus an image-captioning vocabulary with <unk> assigned index 3.
- What you can do after
- Builds token-index dictionaries that safely encode missing vocabulary entries for transformer inputs.
Read how Build a Text-to-Image Generator (from Scratch) explains out-of-vocabulary word →
Deep Learning with Python, Third Edition
A special unknown-token marker used when a tokenizer cannot represent a word with known vocabulary items.
- How it is used
- Uses an unknown-token marker to index input terms absent from a text-processing vocabulary.
- How deep it goes
- Explains vocabulary lookup and fallback indexing with a worked sentence, but does not discuss subword tokenization.
- Examples
- Uses a fox-and-dog sentence, a Python vocabulary dictionary, the word “lazy,” and the resulting index sequence.
- What you can do after
- Apply it via a vocabulary lookup that maps missing tokens to a reserved [UNK] index.
Read how Deep Learning with Python, Third Edition explains out-of-vocabulary word →
Also covered in
What to know first
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- tokenA token is a discrete unit of text, such as a word, subword, punctuation mark, or symbol, produced by tokenization and processed by a language model.
- UnicodeUnicode is a standard that assigns code points and properties to characters and defines encoding forms for representing text digitally.
- accuracyAccuracy is the degree to which a result agrees with a specified truth, reference, or criterion, commonly measured as the proportion of evaluated predictions that are correct.
Related concepts
- FastTextaddressed by · FastText is a word-representation model that learns word vectors using character n-grams and composes representations from subword information.
- subword tokenizationaddressed by · Subword tokenization is a method of representing text with tokens that may be whole words or smaller word fragments, allowing unfamiliar words to be encoded as combinations of units.
- tokenkind of · A token is a discrete unit of text, such as a word, subword, punctuation mark, or symbol, produced by tokenization and processed by a language model.
- tokenizerused by · A tokenizer is a text-processing component or procedure that divides text into discrete tokens and may map them to identifiers according to specified rules or a vocabulary.
- promptalternative to · A prompt is text or other input supplied to an AI model to provide instructions, context, or other information that guides its response or generated output.
Questions about out-of-vocabulary word
What is out-of-vocabulary word?
An out-of-vocabulary word or text unit is absent from a tokenizer’s or model’s recognized vocabulary and may be represented by a special unknown-token marker.
Which Manning books explain out-of-vocabulary word?
Natural Language Processing in Action, Second Edition, LLMs in Production, Learn Generative AI with PyTorch, Deep Learning with R, Third Edition, How Large Language Models Work, Build a Large Language Model (From Scratch), Build a Text-to-Image Generator (from Scratch) and Deep Learning with Python, Third Edition.
What should I know before learning out-of-vocabulary word?
Manning books that teach out-of-vocabulary word usually introduce language model, token, Unicode and accuracy first.
What is the difference between out-of-vocabulary word and prompt?
An out-of-vocabulary word or text unit is absent from a tokenizer’s or model’s recognized vocabulary and may be represented by a special unknown-token marker. A prompt is text or other input supplied to an AI model to provide instructions, context, or other information that guides its response or generated output.
What is out-of-vocabulary word a kind of?
Out-of-vocabulary word is a kind of token. A token is a discrete unit of text, such as a word, subword, punctuation mark, or symbol, produced by tokenization and processed by a language model.
What is out-of-vocabulary word also called?
Out-of-vocabulary word is also called unknown token or [UNK].
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.







