liveBook · Manning

FDD

As explained in Rearchitecting LLMs

FDD is an advanced knowledge-distillation technique that transfers feature-distribution information from a teacher model to a student model.

FDD is an advanced knowledge-distillation technique that aligns dynamics between consecutive layers, transferring feature-distribution information from a teacher to a student. Set delta to activate it in the compound loss; delta=0 disables it. section 6.5.2

Balance the compound-loss weights when enabling FDD:

  • Keep alpha, beta, gamma, and delta summing to 1.0; for example, delta=0.1 and gamma=0.2 allow alpha=0.35 and beta=0.35. section 6.5.2
  • With fewer than 10,000 samples, FDD and Skew KLD show no measurable advantage over basic distillation. Advanced KD begins to show a consistent perplexity advantage at 15,000 or more samples, growing beyond 30,000. section 6.5.2

Reduced-data experiment

Test FDD with delta=0.1–0.2 and Skew KLD with skew_alpha=0.1 using 15,000 Cosmopedia samples, then compare capability retention and perplexity against the 40,000-sample baseline. section 6.7

How Rearchitecting LLMs covers it

How deep it goes
Defines delta activation and weight balancing but does not detail the feature-alignment computation.
Examples
Tests delta values of 0.1–0.2 with Skew KLD on 15,000 Cosmopedia samples against a 40,000-sample baseline.
What you can do after
Configure FDD by setting delta and rebalancing the compound-loss weights to sum to 1.0.
How it connects
Connects to knowledge distillation, compound loss, Skew KLD, and layer pruning.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.