A special token is a reserved control token that marks sequence boundaries or conveys instructions about behavior, structure, or formatting. It gives developers a way to shape how an LLM interprets and produces a prompt sequence. section 5.1.1 section 5.6
Use special tokens to:
- Mark transitions between sequence content, such as the beginning or end of a prompt. section 5.1.1
- Identify conversational roles, such as user and assistant messages. section 5.1.1
- Communicate rules or instructions that guide model behavior and output. section 5.6
A marked conversation sequence
A sequence can use markers such as <|startoftext|>, <|user|>, <|assistant|>, and <|endoftext|> around a question and answer. These markers make the sequence structure explicit before the model processes it. section 5.1.1
<|startoftext|> <|user|> What is the capital of France? <|assistant|> The capital of France is Paris. <|endoftext|>
Inspect the concrete sequence of start, role, and end markers used to illustrate control-token formatting. section 5.1.1
