7 Knowledge distillation: Capturing frontier model intelligence
This chapter covers
- Building a black-box distillation pipeline from teacher data generation through student training
- Filtering teacher outputs to remove degenerate or low-quality responses
- Evaluating distillation quality with three-way comparisons (base, teacher, student)
- Understanding when forward vs. reverse Kullback-Leibler (KL) divergence matters
- Testing whether safety alignment transfers from teacher to student (it does not)
Your chapter 6 SFT model answers IT support questions well, but the frontier API you tested during project scoping (GPT-6 Astra, Claude Opus 5, or Gemini 3.1 Pro) answers them better. The catch: that API costs $5,000 per month at your current query volume, and every request sends proprietary data off-network. What if you could capture 80 to 90% of the frontier model’s quality in a smaller, locally run model at a fraction of the cost?
Continuing the chapter 5 example, picture distilling that Contoso support assistant rather than a generic model. The question isn't only whether the student maintains overall accuracy, but whether it still preserves the house answer format and domain behavior, because those specifics are the easiest to lose when you compress a model. As before, the hands-on uses the reproducible IT support data, while the lesson applies to any internal assistant you have already specialized in.