Special token
A special token is a reserved vocabulary item that conveys structural, control, boundary, error, or other auxiliary information rather than ordinary content.
How Large Language Models Work
A special token is a reserved token that conveys control, boundary, error, modality, or other auxiliary information rather than ordinary word content.
- How it is used
- Uses reserved tokens to communicate boundaries, errors, modalities, system data, and domain-specific information.
- How deep it goes
- Defines several roles with examples, but stops short of detailing token IDs, embeddings, or implementation code.
- Examples
- Shows [UNK], [SYSTM], text-image boundary markers, domain tokens, digit tokens, and a shared number token.
- What you can do after
- Evaluate how reserved-token choices affect input boundaries, modality handling, unknown symbols, and numerical behavior.
Read how How Large Language Models Work explains special token →
Build AI-Enhanced Web Apps
A control token is a reserved token that marks boundaries or conveys instructions about the structure, behavior, or formatting of a language-model sequence.
- How it is used
- Guides model behavior and marks structure within prompts and conversational sequences.
- How deep it goes
- Defines control tokens and shows a formatted sequence, but stops short of implementation details for creating or registering them.
- Examples
- Uses start-of-text, user, assistant, and end-of-text markers in a formatted question-and-answer sequence.
- What you can do after
- Apply it by structuring prompt sequences with markers that communicate boundaries, roles, rules, or output behavior.
Read how Build AI-Enhanced Web Apps explains special token →
LLMs in Production
A special token is a reserved token with a syntactic or control function in tokenized model input.
- How it is used
- Uses reserved syntactic markers when training custom tokenizers for language-model data.
- How deep it goes
- Defines the parameter and shows its configuration, but defers the internal operation of BPE and SentencePiece.
- Examples
- Uses <s>, <pad>, </s>, <unk>, and <mask> in a custom ByteLevelBPE training configuration.
- What you can do after
- Build it from scratch by configuring special tokens while training a custom BPE or SentencePiece tokenizer.
Reinforcement Learning from Human Feedback
A special token is a reserved token that marks structural or control information in a model's input or output sequence.
- How it is used
- Structures instruction-tuned conversations and coordinates model generation with externally executed tool calls.
- How deep it goes
- Explains token placement and pipeline behavior, but defers tokenizer-specific implementation and provider-format details.
- Examples
- Uses role delimiters, assistant-generation prompts, end-of-sequence markers, function-call tags, and Fibonacci code-execution outputs.
- What you can do after
- Operate chat templates and tool-use pipelines by recognizing message boundaries, generation prompts, tool calls, and injected results.
Read how Reinforcement Learning from Human Feedback explains special token →
Effective Conversational AI
A special token is a reserved token with a control or structural meaning in a language-model input or output.
- How it is used
- Separates prompt instructions, conversation elements, examples, and requested outputs for language-model summarization.
- How deep it goes
- Demonstrates marker placement and output effects, but does not explain tokenization internals or model training.
- Examples
- Uses one-shot claim-status transcripts marked with instruction, transcript, and summary delimiters.
- What you can do after
- Diagnose/debug with it when ambiguous prompt delimiters cause poor summaries or weak separation between instructions and conversation.
Read how Effective Conversational AI explains special token →
Related concepts
- end of text tokenincludes · An end-of-text token is a special token marking the boundary or termination of a text sequence, document, or generated output.
- tool usageused by · Tool use is an AI system's invocation of external software functions, services, or other capabilities to obtain information or perform actions.
- chat templateused by · A chat template is a formatting specification that serializes conversational messages into the structured input representation expected by a language model.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.