FDD is an advanced knowledge-distillation technique that aligns dynamics between consecutive layers, transferring feature-distribution information from a teacher to a student. Set delta to activate it in the compound loss; delta=0 disables it. section 6.5.2
Balance the compound-loss weights when enabling FDD:
- Keep alpha, beta, gamma, and delta summing to 1.0; for example, delta=0.1 and gamma=0.2 allow alpha=0.35 and beta=0.35. section 6.5.2
- With fewer than 10,000 samples, FDD and Skew KLD show no measurable advantage over basic distillation. Advanced KD begins to show a consistent perplexity advantage at 15,000 or more samples, growing beyond 30,000. section 6.5.2
Reduced-data experiment
Test FDD with delta=0.1–0.2 and Skew KLD with skew_alpha=0.1 using 15,000 Cosmopedia samples, then compare capability retention and perplexity against the 40,000-sample baseline. section 6.7
