Model compression reduces a machine-learning model’s size or computational requirements while attempting to preserve its behavior. Knowledge distillation compresses a large teacher into one or more smaller student models, transferring useful capabilities without retaining the teacher’s full deployment cost. section 1.2.3 section 8.1.1
Compression addresses several concrete deployment constraints:
- Reduces memory requirements: DeepSeek-R1’s 671 billion parameters require approximately 1,342 GB in FP16, even though only 37 billion activate per token. section 8.1.1
- Expands deployment targets from multi-node H100 clusters toward single GPUs, laptops, offline tools, and other constrained environments. section 8.1.1
- Preserves substantial reasoning ability through smaller checkpoints, including 1.5B, 7B, 8B, 14B, 32B, and 70B models. section 1.2.3
- Accepts capability trade-offs: a 1.5B student excels at mathematics but performs poorly on coding, while distillation cannot exceed the teacher’s capability frontier. section 8.9
DeepSeek-R1 distillation
A teacher generated 800,000 chain-of-thought traces through rejection sampling, retaining verified and well-formatted answers across mathematics, coding, STEM, logic, and general tasks. Supervised fine-tuning on these traces produced DeepSeek-R1-Qwen-1.5B, which scored 28.9% on AIME 2024 versus GPT-4o’s 9.3%. The result compressed reasoning capability by approximately 450 times in parameter count. On the same architecture, distillation reached 72.6% on AIME compared with 47.0% for reinforcement learning alone. section 8.9
