liveBook · Manning

Foundation model

As explained in Build a DeepSeek Model (From Scratch)

A base model is a pretrained model whose existing parameters and capabilities provide the foundation for subsequent adaptation or distillation.

A foundational model is a pretrained model whose existing parameters and capabilities provide the starting point for adaptation or knowledge distillation. Its prior knowledge affects how effectively a student absorbs new reasoning patterns. section 8.7.3 section 8.5.4

Choose the foundation according to the target capability:

  • Use a math-specialized base when mathematical reasoning is central; Qwen2.5-Math gives smaller distilled models a head start. section 8.5.4
  • Use a stronger general-purpose base when broader capability matters; Qwen2.5 bases outperform larger Llama bases on the reported math benchmark. section 8.7.3
  • Treat architecture family and model variant as strategic choices, not interchangeable implementation details. section 8.7.3 section 8.5.4

Distillation lineup

The six-model lineup pairs Qwen2.5 and Llama 3 foundations with distilled training: Qwen 1.5B and 7B use math-specialized bases, while Qwen 14B, Qwen 32B, Llama 8B, and Llama 70B use general-purpose bases. section 8.5.4

Figure 8.20 DeepSeek’s six distilled models span two architecture families: Qwen2.5 (1.5B, 7B, 14B, 32B) and Llama-3 (8B, 70B). The 1.5B and 7B Qwen models use math-specialized base models, while larger models use general-purpose bases.

Compare the model families, parameter sizes, base types, and initial learning rates used in the distillation lineup. section 8.5.4

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Compares specialized and general bases through model variants and benchmarks, without detailing pretraining internals.
Examples
Uses Qwen2.5-Math, Qwen2.5, Llama 3, and six DeepSeek distilled model variants as concrete comparisons.
What you can do after
Evaluate candidate foundations against target capabilities and expected distilled-model performance.
How it connects
Connects pretrained knowledge, knowledge distillation, model scaling, architecture families, and mathematical reasoning.

Also explained in

Compare how 30 books explain foundation model →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.