An out-of-vocabulary word is missing from the model’s token dictionary. Mapping it to a special UNK token gives it an index the model can process, instead of causing a lookup failure. section 12.2.1 section 10.1.1
Fallback versus subword tokens
If “technology” is absent from a word-level dictionary, looking it up without UNK can crash; adding UNK allows it to be encoded. By contrast, a tokenizer may split “French” into “fr” and “ench,” so the word need not be represented as one vocabulary entry. section 12.2.1 section 10.1.1
Listing 12.4 Mapping tokens to indexes
from collections import Counter
word_counts=Counter(text_tokenized)
words=sorted(word_counts, key=word_counts.get,
reverse=True)
words.append("UNK") #1
text_length=len(text_tokenized)
ntokens=len(words) #2
print(f"the text contains {text_length} words")
print(f"there are {ntokens} unique tokens")
word_to_int={v:k for k,v in enumerate(words)} #3
int_to_word={v:k for k,v in word_to_int.items()} #4
print({k:v for k,v in word_to_int.items() if k in words[:10]})
print({k:v for k,v in int_to_word.items() if v in words[:10]})
See how UNK is added to the token list and tokens are mapped to integer indexes. section 12.2.1
