liveBook · Manning

Scaling law

As explained in Build a DeepSeek Model (From Scratch)

A scaling law is an empirical relationship describing how a model's performance or other properties change as resources such as model size increase.

A scaling law is an empirical relationship showing how model performance changes as resources such as model size increase. Distilled-model performance can improve smoothly across sizes, rising from 28.9% at 1.5B parameters to 72.6% at 32B on AIME 2024. section 8.7.3

Scaling is not determined by parameter count alone. Base-model knowledge and architecture interact with distillation: Qwen2.5-Math bases outperform similarly sized Llama bases, while Qwen-32B exceeds Llama-70B. Open questions examine teacher size, student size, dataset size, diminishing returns, and power-law behavior. section 8.7.3 section 8.8.4

Distilled-model comparison

On AIME 2024, Qwen-7B scores 55.5% versus Llama-8B at 50.4%; Qwen-32B reaches 72.6%, slightly above Llama-70B at 70.0%. The comparison shows why selecting a capable base model can matter more than adding parameters alone. section 8.7.3

Figure 8.27 Performance (AIME 2024) scales smoothly with model size for distilled models. Qwen-Math bases consistently outperform Llama bases at similar sizes, showing that the base model’s pre-existing capabilities interact with distillation quality.

Use the performance plot to compare smooth size scaling and the persistent advantage of Qwen-Math bases over Llama bases. section 8.7.3

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Explains observed size trends and base-model interactions, then identifies teacher, student, and dataset scaling as open questions.
Examples
Compares Qwen2.5-Math, Qwen2.5, and Llama-3 distilled models using AIME 2024 performance across parameter sizes.
What it assumes
Assumes familiarity with knowledge distillation, teacher and student models, parameter size, and benchmark performance.
What you can do after
Evaluate how model size and base-model capabilities affect distilled-model performance.
How it connects
Connects model size with knowledge distillation, teacher size, student size, dataset size, and base-model pretraining.

Also explained in

Compare how 4 books explain scaling law →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.