A foundational model is a pretrained model whose existing parameters and capabilities provide the starting point for adaptation or knowledge distillation. Its prior knowledge affects how effectively a student absorbs new reasoning patterns. section 8.7.3 section 8.5.4
Choose the foundation according to the target capability:
- Use a math-specialized base when mathematical reasoning is central; Qwen2.5-Math gives smaller distilled models a head start. section 8.5.4
- Use a stronger general-purpose base when broader capability matters; Qwen2.5 bases outperform larger Llama bases on the reported math benchmark. section 8.7.3
- Treat architecture family and model variant as strategic choices, not interchangeable implementation details. section 8.7.3 section 8.5.4
Distillation lineup
The six-model lineup pairs Qwen2.5 and Llama 3 foundations with distilled training: Qwen 1.5B and 7B use math-specialized bases, while Qwen 14B, Qwen 32B, Llama 8B, and Llama 70B use general-purpose bases. section 8.5.4
Figure 8.20 DeepSeek’s six distilled models span two architecture families: Qwen2.5 (1.5B, 7B, 14B, 32B) and Llama-3 (8B, 70B). The 1.5B and 7B Qwen models use math-specialized base models, while larger models use general-purpose bases.

Compare the model families, parameter sizes, base types, and initial learning rates used in the distillation lineup. section 8.5.4





























