An out-of-vocabulary (OOV) word is a word a pipeline has not given its own place in the vocabulary. A word-based pipeline must decide which tokens to retain; otherwise its vocabulary can grow to tens of thousands of words. Tokens left out may be ignored or grouped under one generic token. section 2.2 section 3.1
What an unknown token preserves—and loses
A pipeline could count an unusual word such as “supercalifragilistic” under <OOV>. The count records that an excluded token occurred, but the pipeline can no longer compute a meaning for that individual word. This matters when deciding which words deserve their own entries in a bag-of-words vocabulary. section 3.1
Tokenization changes how often this problem arises:
- Word tokens can produce a large vocabulary and require a decision about which words count. section 2.2
- Byte- or character-based pipelines use a smaller vocabulary, helping them handle unseen text without many meaningless OOV tokens. section 2.2
- A subword tokenizer keeps individual characters available, so it need not use an OOV token when new text contains no new characters. section 2.2.1
Do not assume every <unk> marks a naturally unseen word. In one PyTorch WikiText2 dataset, <unk> also replaces tokens as a mask. A word-based RNN trained as though these markers were ordinary words can waste effort trying to learn their meaning; using raw text without that masking is one way to avoid this source of error. section 8.5
>>> num_eos = sum([vocab.idx2word[i] == '<eos>' for i in
corpus.train.numpy()])
>>> num_eos
36718
>>> num_unk = sum([vocab.idx2word[i] == '<unk>' for i in
corpus.train.numpy()])
>>> num_unk
54625
>>> num_normal = sum([
... vocab.idx2word[i] not in ('<unk>', '<eos>')
... for i in corpus.train.numpy()])
>>> num_normal
1997285
>>> num_unk / (num_normal + num_eos + num_unk)
0.0261...
Count <unk> alongside ordinary and end-of-sequence tokens to inspect its frequency in the training data; this calculation puts the marker at about 2.6% of tokens. section 8.5
