liveBook · Manning

Model compression

As explained in Build a DeepSeek Model (From Scratch)

Model compression is the reduction of a machine-learning model's size or computational requirements while attempting to preserve its behavior.

Model compression reduces a machine-learning model’s size or computational requirements while attempting to preserve its behavior. Knowledge distillation compresses a large teacher into one or more smaller student models, transferring useful capabilities without retaining the teacher’s full deployment cost. section 1.2.3 section 8.1.1

Compression addresses several concrete deployment constraints:

  • Reduces memory requirements: DeepSeek-R1’s 671 billion parameters require approximately 1,342 GB in FP16, even though only 37 billion activate per token. section 8.1.1
  • Expands deployment targets from multi-node H100 clusters toward single GPUs, laptops, offline tools, and other constrained environments. section 8.1.1
  • Preserves substantial reasoning ability through smaller checkpoints, including 1.5B, 7B, 8B, 14B, 32B, and 70B models. section 1.2.3
  • Accepts capability trade-offs: a 1.5B student excels at mathematics but performs poorly on coding, while distillation cannot exceed the teacher’s capability frontier. section 8.9

DeepSeek-R1 distillation

A teacher generated 800,000 chain-of-thought traces through rejection sampling, retaining verified and well-formatted answers across mathematics, coding, STEM, logic, and general tasks. Supervised fine-tuning on these traces produced DeepSeek-R1-Qwen-1.5B, which scored 28.9% on AIME 2024 versus GPT-4o’s 9.3%. The result compressed reasoning capability by approximately 450 times in parameter count. On the same architecture, distillation reached 72.6% on AIME compared with 47.0% for reinforcement learning alone. section 8.9

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Explains memory costs, teacher-student transfer, dataset construction, benchmarks, and capability limits, but does not provide implementation code.
Examples
Uses DeepSeek-R1 checkpoints, 800K reasoning traces, AIME and MATH scores, coding benchmarks, and laptop, GPU, and rural-clinic scenarios.
What it assumes
Assumes familiarity with parameters, inference, mixture-of-experts, KV cache, and reinforcement learning while grounding compression through concrete costs and results.
What you can do after
Evaluate model-size, memory, benchmark, and capability trade-offs when selecting compressed models for constrained deployment.
How it connects
Connects model compression with knowledge distillation, supervised fine-tuning, reinforcement learning, mixture-of-experts, FP16, and KV cache memory.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.