Skew KL divergence (Skew KLD) modifies the logits loss used to distill a pruned student model. Instead of comparing teacher and student distributions directly, it compares the student against a mixture of both. This can soften the penalty when the student’s predictions are far from the teacher’s, helping recovery from aggressive pruning. section 6.4 section 6.4.1
In the implementation, the mixture is controlled by `skew_alpha`:
- Set `skew_alpha=0.1` to mix 10% student probabilities with 90% teacher probabilities. The mixture softens the early training penalty; as the student improves, the target becomes stricter. section 6.4.1
- Use the mixture probabilities in the logits loss and add a small constant such as `1e-9` before taking their logarithm to avoid log-zero numerical issues. section 6.4.1
- Set `skew_alpha=0` to disable Skew KLD. For fewer than 10,000 training samples, advanced distillation showed no measurable advantage over basic distillation; results favored advanced techniques more consistently with 15,000 or more samples. section 6.5.2
A pruned student’s token prediction
After “Paris is the capital of,” suppose the teacher assigns “France” 70% probability while the student assigns it 30%. Mixing their distributions makes the target less demanding than the teacher’s prediction alone at the start of training. section 6.4.1
Figure 6.7 Target distribution comparison. Left panel: Standard KLD with teacher targets (solid) and student predictions (diagonal hatching). Right panel: Skew KLD adds hybrid targets (dotted) computed as 90% teacher + 10% student.
Compare the teacher and student targets with the hybrid target formed from 90% teacher and 10% student probabilities. section 6.4.1
