A special token is a reserved token that adds control or auxiliary information instead of ordinary word content. Tokenizers may use it when a symbol cannot be processed, when the input changes source or modality, or when the model must distinguish system instructions from user data. section 2.2.3
Common roles include:
- [UNK] marks an unknown symbol when the tokenizer fails to process it correctly. section 2.2.3
- [SYSTM] distinguishes built-in prompts from user-entered data and can mark stylistic information. section 2.2.3
- Multimodal models use distinct tokens to signal a switch between text bytes and image bytes. section 2.2.3
- A domain-specific token can preserve an important word or concept that ordinary subword construction might split awkwardly. section 2.2.3
A boundary in the input stream
A prompt can contain a system instruction followed by user-entered text. A reserved system token marks that boundary so the model receives information about where each part came from. In a text-and-image prompt, another reserved token can mark the transition from text data to image data. section 2.2.3
Numbers show why token design matters. A tokenizer may represent a number inconsistently across digit positions, making arithmetic harder. Some proposed approaches use individual digits; another replaces each number with a shared number token and represents its value separately. section 6.2.2
