Special token
A special token is a reserved vocabulary item that conveys structural, control, boundary, or other auxiliary information rather than ordinary content.
How Large Language Models Work
A special token is a reserved token that conveys control, boundary, error, modality, or other auxiliary information rather than ordinary word content.
- How it is used
- Uses reserved tokens to communicate boundaries, errors, modalities, system data, and domain-specific information.
- How deep it goes
- Defines several roles with examples, but stops short of detailing token IDs, embeddings, or implementation code.
- Examples
- Shows [UNK], [SYSTM], text-image boundary markers, domain tokens, digit tokens, and a shared number token.
- What you can do after
- Evaluate how reserved-token choices affect input boundaries, modality handling, unknown symbols, and numerical behavior.
Read how How Large Language Models Work explains special token →
Build AI-Enhanced Web Apps
A control token is a reserved token that marks boundaries or conveys instructions about the structure, behavior, or formatting of a language-model sequence.
- How it is used
- Guides model behavior and marks structure within prompts and conversational sequences.
- How deep it goes
- Defines control tokens and shows a formatted sequence, but stops short of implementation details for creating or registering them.
- Examples
- Uses start-of-text, user, assistant, and end-of-text markers in a formatted question-and-answer sequence.
- What you can do after
- Apply it by structuring prompt sequences with markers that communicate boundaries, roles, rules, or output behavior.
Read how Build AI-Enhanced Web Apps explains special token →
LLMs in Production
A special token is a reserved token with a syntactic or control function in tokenized model input.
- How it is used
- Uses reserved syntactic markers when training custom tokenizers for language-model data.
- How deep it goes
- Defines the parameter and shows its configuration, but defers the internal operation of BPE and SentencePiece.
- Examples
- Uses <s>, <pad>, </s>, <unk>, and <mask> in a custom ByteLevelBPE training configuration.
- What you can do after
- Build it from scratch by configuring special tokens while training a custom BPE or SentencePiece tokenizer.
Reinforcement Learning from Human Feedback
A special token is a reserved token that marks structural or control information in a model's input or output sequence.
- How it is used
- Structures instruction-tuned conversations and coordinates model generation with externally executed tool calls.
- How deep it goes
- Explains token placement and pipeline behavior, but defers tokenizer-specific implementation and provider-format details.
- Examples
- Uses role delimiters, assistant-generation prompts, end-of-sequence markers, function-call tags, and Fibonacci code-execution outputs.
- What you can do after
- Operate chat templates and tool-use pipelines by recognizing message boundaries, generation prompts, tool calls, and injected results.
Read how Reinforcement Learning from Human Feedback explains special token →
Effective Conversational AI
A special token is a reserved token with a control or structural meaning in a language-model input or output.
- How it is used
- Separates prompt instructions, conversation elements, examples, and requested outputs for language-model summarization.
- How deep it goes
- Demonstrates marker placement and output effects, but does not explain tokenization internals or model training.
- Examples
- Uses one-shot claim-status transcripts marked with instruction, transcript, and summary delimiters.
- What you can do after
- Diagnose/debug with it when ambiguous prompt delimiters cause poor summaries or weak separation between instructions and conversation.
Read how Effective Conversational AI explains special token →
What to know first
- tokenA token is a discrete unit of text, such as a word, subword, punctuation mark, or symbol, produced by tokenization and processed by a language model.
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- context windowA context window is a bounded span of sequence elements or contextual information that a model or processing operation can consider together.
- GPT-4GPT-4 is a multimodal large language model developed by OpenAI that processes text and image inputs and generates text.
Related concepts
- end-of-sequence tokenincludes · An end-of-sequence token is a special token that marks the end of a token sequence or document and may signal that generation should stop.
- reflection tokenincludes · A reflection token is a special token through which a language model signals judgments about retrieval needs or the usefulness of generated content.
- tool useused by · Tool use is the capability of an AI system or agent to request external software operations through defined interfaces to obtain information or perform actions.
- chat templateused by · A chat template is a formatting specification that converts conversational messages into the input representation expected by a language model.
Questions about special token
What is special token?
A special token is a reserved vocabulary item that conveys structural, control, boundary, or other auxiliary information rather than ordinary content.
Which Manning books explain special token?
How Large Language Models Work, Build AI-Enhanced Web Apps, LLMs in Production, Reinforcement Learning from Human Feedback and Effective Conversational AI.
What should I know before learning special token?
Manning books that teach special token usually introduce token, language model, context window and GPT-4 first.
What kinds of special token are there?
Kinds of special token include end-of-sequence token, reflection token, stop parameter and tool-call token.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.




