An out-of-vocabulary word is a token absent from the model’s vocabulary. A reserved unknown marker—UNK or <unk>—stands in for it, allowing the token sequence to become an integer sequence that the model can accept. section 2.2.1 section 4.2.2
Handle an unknown token through the vocabulary pipeline:
- Look up each token with a fallback to UNK or <unk> when the token is missing. section 4.2.2
- Use the resulting index sequence as model input; reverse dictionaries can reconstruct known tokens, while unknown entries remain represented by the marker. section 2.2.1
Caption vocabulary
An image-captioning vocabulary assigns <pad> to 0, <start> to 1, <end> to 2, and <unk> to 3. A caption token missing from word2idx therefore becomes index 3. section 4.2.2
