A logit is an unnormalized model score. Applying softmax converts logits into probabilities, which can then be compared during knowledge distillation. section 2.5
In logits-only distillation, the training cycle is:
- Pass a text batch to the teacher and collect its logits. section 2.5
- Pass the same batch to the student and collect its logits. section 2.5
- Convert both outputs to probability distributions, measure their difference with KL divergence, and update the student’s weights. section 2.5
Listing 2.3 Step 2: Getting the teacher's answer(logits)
with torch.no_grad():
teacher_outputs = teacher_model(
input_ids=input_ids,
attention_mask=attention_mask
)
teacher_logits = teacher_outputs.logits
See how a teacher model produces logits from input IDs and an attention mask. section 2.5























