Tokenization prepares text for model training by rendering prompts and answers, converting them into token IDs, and recording boundaries needed by the training loss. section 8.4 section 8.4.2
A practical tokenization pipeline:
- Render and tokenize the prompt. section 8.4.2
- Format and tokenize the teacher answer. section 8.4.2
- Concatenate prompt and answer IDs, append an end-of-sequence token, and store prompt length. section 8.4.2
- Decode a processed example to verify chat formatting, reasoning tags, and termination. section 8.4.2
Inspecting a distilled math example
A formatted sample contains a user chat prompt, an assistant answer wrapped with <think></think>, a final boxed result, and <|im_end|> termination. section 8.4.2
Figure 8.7 Bringing the loaded dataset into a format suitable for model training by understanding the tokenizer, tokenizing the examples, and filtering and splitting the dataset
Use the workflow figure to trace dataset loading, tokenizer use, tokenization, filtering, and splitting before training. section 8.4
