liveBook · Manning

Skew KLD

As explained in Rearchitecting LLMs

Skew KL divergence is an asymmetric divergence that compares a probability distribution with a mixture of distributions, and can be used as a distillation objective.

Skew KL divergence (Skew KLD) modifies the logits loss used to distill a pruned student model. Instead of comparing teacher and student distributions directly, it compares the student against a mixture of both. This can soften the penalty when the student’s predictions are far from the teacher’s, helping recovery from aggressive pruning. section 6.4 section 6.4.1

In the implementation, the mixture is controlled by `skew_alpha`:

  • Set `skew_alpha=0.1` to mix 10% student probabilities with 90% teacher probabilities. The mixture softens the early training penalty; as the student improves, the target becomes stricter. section 6.4.1
  • Use the mixture probabilities in the logits loss and add a small constant such as `1e-9` before taking their logarithm to avoid log-zero numerical issues. section 6.4.1
  • Set `skew_alpha=0` to disable Skew KLD. For fewer than 10,000 training samples, advanced distillation showed no measurable advantage over basic distillation; results favored advanced techniques more consistently with 15,000 or more samples. section 6.5.2

A pruned student’s token prediction

After “Paris is the capital of,” suppose the teacher assigns “France” 70% probability while the student assigns it 30%. Mixing their distributions makes the target less demanding than the teacher’s prediction alone at the start of training. section 6.4.1

Figure 6.7 Target distribution comparison. Left panel: Standard KLD with teacher targets (solid) and student predictions (diagonal hatching). Right panel: Skew KLD adds hybrid targets (dotted) computed as 90% teacher + 10% student.

Compare the teacher and student targets with the hybrid target formed from 90% teacher and 10% student probabilities. section 6.4.1

How Rearchitecting LLMs covers it

How deep it goes
Implements the mixture-based loss and numerical safeguard, but does not derive the divergence formally.
Examples
Uses a France next-token probability comparison and reports recovery results across training-set sizes.
What it assumes
Assumes teacher and student logits and a standard KL logits loss; explains the mixture adjustment rather than introducing distillation from scratch.
What you can do after
Apply the loss in a distillation workflow by configuring `skew_alpha` and mixing teacher and student probabilities.
How it connects
Connects KL divergence to knowledge distillation, model pruning, and Feature Dynamics Distillation.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.