A tokenizer prepares language-model input by splitting text into tokens, converting them to numerical IDs, and decoding generated IDs back into readable text. It sits outside the LLM but participates in every prompt, reasoning trace, answer, and generation step. section 1.2 section 2.10
Use the tokenizer through this implementation workflow:
- Load tokenizer files and initialize Qwen3Tokenizer. section 2.4
- Encode a prompt into token IDs before passing it to the model. section 2.4
- Decode token IDs to inspect reconstructed text or generated output. section 2.4
- Use byte pair encoding to represent common and rare words through full-word and subword units. section 2.4
The Qwen3 reasoning tokenizer adds support for <think>...</think> tokens. With chat formatting enabled, it wraps prompts with user and assistant markers; answer text can disable wrapping before concatenation. section 8.4.1
Round-trip encoding
Encoding "Explain large language models." produces six IDs: 840→"Ex", 20772→"plain", 3460→" large", 4128→" language", 4119→" models", and 13→".". Decoding them reconstructs the original prompt. section 2.4
Training sequence with reasoning
Encode a wrapped prompt, encode an answer with chat_wrapped=False, then concatenate prompt IDs, answer IDs, and the end-of-sequence ID. The resulting sequence contains the prompt, reasoning trace, final answer, and closing marker. section 8.4.1
Figure 2.5 A simplified illustration of how an LLM receives input data and generates output. The user-provided text is tokenized into IDs using the tokenizer’s encode method, which are then processed by the LLM to generate output token IDs. These are decoded back into human-readable text using the tokenizer’s decode method.
Trace the complete input-output path from user text through encode, model processing, generated IDs, and decode. section 2.4
