A scaling law is an empirical relationship showing how model performance changes as resources such as model size increase. Distilled-model performance can improve smoothly across sizes, rising from 28.9% at 1.5B parameters to 72.6% at 32B on AIME 2024. section 8.7.3
Scaling is not determined by parameter count alone. Base-model knowledge and architecture interact with distillation: Qwen2.5-Math bases outperform similarly sized Llama bases, while Qwen-32B exceeds Llama-70B. Open questions examine teacher size, student size, dataset size, diminishing returns, and power-law behavior. section 8.7.3 section 8.8.4
Distilled-model comparison
On AIME 2024, Qwen-7B scores 55.5% versus Llama-8B at 50.4%; Qwen-32B reaches 72.6%, slightly above Llama-70B at 70.0%. The comparison shows why selecting a capable base model can matter more than adding parameters alone. section 8.7.3
Figure 8.27 Performance (AIME 2024) scales smoothly with model size for distilled models. Qwen-Math bases consistently outperform Llama bases at similar sizes, showing that the base model’s pre-existing capabilities interact with distillation quality.

Use the performance plot to compare smooth size scaling and the persistent advantage of Qwen-Math bases over Llama bases. section 8.7.3



