liveBook · Manning

Out-of-vocabulary word

As explained in Deep Learning with Python, Third Edition

A special unknown-token marker used when a tokenizer cannot represent a word with known vocabulary items.

An out-of-vocabulary word is a token absent from the vocabulary used to index text. A special [UNK] token gives it a stable fallback index instead of leaving the input unrepresentable. section 14.2

Mapping an unseen word

Suppose the vocabulary assigns [UNK] to 0 and contains the words in “The quick brown fox jumped over the dog.” The sentence with “lazy” produces [0, 2, 3, 4, 5, 6, 1, 0, 7, 8]: “lazy” becomes 0 because it is not known. “The” also falls back to 0 in this lowercase vocabulary. section 14.2

vocabulary = {
    "[UNK]": 0,
    "the": 1,
    "quick": 2,
    "brown": 3,
    "fox": 4,
    "jumped": 5,
    "over": 6,
    "dog": 7,
    ".": 8,
}
words = split_words("The quick brown fox jumped over the lazy dog.")
indices = [vocabulary.get(word, 0) for word in words]

Inspect the vocabulary lookup and see how missing tokens receive the [UNK] index; compare the resulting sequence in b7. section 14.2

How Deep Learning with Python, Third Edition covers it

How deep it goes
Explains vocabulary lookup and fallback indexing with a worked sentence, but does not discuss subword tokenization.
Examples
Uses a fox-and-dog sentence, a Python vocabulary dictionary, the word “lazy,” and the resulting index sequence.
What it assumes
Assumes token splitting and integer indexing while introducing vocabulary-based representation from the basics.
What you can do after
Apply it via a vocabulary lookup that maps missing tokens to a reserved [UNK] index.
How it connects
Connects unknown tokens to tokenization, vocabularies, indices, text preprocessing, and standardization.
Cover of Deep Learning with Python, Third Edition
liveBookmanning.com
Read Deep Learning with Python, Third Edition on liveBook Get the book at manning.com

Also explained in

Compare how 8 books explain out-of-vocabulary word →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Deep Learning with Python, Third Edition. The section links point to the parts of the book it draws on.