An unknown token is a vocabulary placeholder for text that a tokenizer cannot recognize. SimpleTokenizerV2 replaces out-of-vocabulary words with <|unk|>, allowing encoding to continue. section 2.4
The tokenizer adds <|unk|> to its vocabulary, maps it to an integer ID, and restores the marker during decoding. This preserves token sequence structure but loses the original word.
Byte pair encoding avoids this placeholder: it decomposes unfamiliar words into subwords or characters, so the tokenizer can represent arbitrary input and reconstruct it. section 2.5
SimpleTokenizerV2 output
Encoding “Hello, do you like tea? <|endoftext|> In the sunlit terraces of the palace.” produces <|unk|> for unseen words such as “Hello” and “palace,” while preserving the document separator.
