UNK is a special token for words or token pieces outside a model’s vocabulary. During tokenization, it replaces an unknown item with a known index so prompts and sentences remain processable. section 12.2.1 section 10.1.1
A vocabulary cannot contain every possible token. Adding UNK prevents a lookup failure when a prompt includes a word such as “technology” that is absent from the dictionary. The token is assigned an index alongside ordinary vocabulary entries and can be recovered through the reverse mapping. section 12.2.1
UNK appears in two related tokenization setups:
- Word-level tokenization appends UNK to a vocabulary of 10,599 tokens, producing 10,600 indexed tokens for the combined novels. section 12.2.1
- A translation dictionary reserves index 1 for UNK and index 0 for PAD, while BOS and EOS mark sentence boundaries. section 10.1.1
- Subword tokenization can split a known word such as “French” into “fr” and “ench”; UNK remains the fallback for tokens not found in the retained dictionary. section 10.1.1
Mapping an unknown token
After tokenization, a lookup such as en_word_dict.get(i, UNK) converts an absent English token to index 1. The reverse dictionary can then represent that index as UNK, keeping the integer sequence valid for a model. section 10.1.1
Using UNK as a generation prompt
Setting prompt="UNK" lets a trained text generator produce text unconditionally. Repeating generation with fixed random seeds yields comparable samples for checking grammatical quality and possible copying from training data. section 12.4.3
Listing 12.4 Mapping tokens to indexes
from collections import Counter
word_counts=Counter(text_tokenized)
words=sorted(word_counts, key=word_counts.get,
reverse=True)
words.append("UNK") #1
text_length=len(text_tokenized)
ntokens=len(words) #2
print(f"the text contains {text_length} words")
print(f"there are {ntokens} unique tokens")
word_to_int={v:k for k,v in enumerate(words)} #3
int_to_word={v:k for k,v in word_to_int.items()} #4
print({k:v for k,v in word_to_int.items() if k in words[:10]})
print({k:v for k,v in int_to_word.items() if v in words[:10]})
The mapping listing shows where UNK is appended, how token-to-index and index-to-token dictionaries are constructed, and how vocabulary size changes. section 12.2.1
