A teacher model guides a student model during knowledge distillation. Its predictions provide a target for a smaller student to imitate, helping recover capabilities lost through pruning. Matching the teacher’s full prediction distribution transfers information about alternatives, not just the highest-probability answer. section 2.5 section 2.8
For each training batch, the teacher and student process the same input IDs and attention mask. The teacher’s forward pass runs without gradient calculation. KL divergence compares the student’s and teacher’s output distributions; backpropagation then updates the student’s weights. section 2.5
Listing 2.5 Step 4: Correcting the attempt
loss = F.kl_div( F.log_softmax(student_logits, dim=-1), F.softmax(teacher_logits, dim=-1), reduction='batchmean' ) loss.backward() #A optimizer.step() #B optimizer.zero_grad() #C
This loss step compares the student’s log probabilities with the teacher’s probabilities and updates the student. section 2.5
Configure the teacher in evaluation mode and freeze its parameters to avoid computing teacher gradients, saving GPU memory and time. Tokenize recovery data in batches and reserve separate validation data: evaluating perplexity on training samples can give an overly optimistic measure of recovery. section 6.1.2
Listing 6.2 Teacher model and tokenizer configuration
MODEL_NAME = "google/gemma-3-270m"
teacher_model = AutoModelForCausalLM.from_pretrained(
MODEL_NAME,
torch_dtype=torch.bfloat16 if torch.cuda.is_available() else torch.float32, #A
device_map="auto" if torch.cuda.is_available() else None
)
teacher_model.eval() #B
for param in teacher_model.parameters():
param.requires_grad = False #C
tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)
if tokenizer.pad_token is None:
tokenizer.pad_token = tokenizer.eos_token
This configuration loads Gemma, freezes the teacher, and supplies a padding token when the tokenizer lacks one. section 6.1.2
Recovering a pruned model
An 18-layer gemma-3-270m model serves as teacher for a 16-layer student created by pruning a copy of it. The recovery experiment trains on 15,000 SlimPajama rows; pruning is relatively light, while recovery training needs substantially more compute and should run on a GPU. section 2.5
Replacing an agent call
A frontier model can serve as both teacher and evaluation bar for a specialized small model replacing one agent sub-task. Its existing outputs on the agent’s inputs define the behavior the replacement must learn and match. section 12.1





