liveBook · Manning

Teacher model

As explained in Rearchitecting LLMs

A teacher model is a model whose outputs or behavior guide the training of a student model during knowledge distillation.

A teacher model guides a student model during knowledge distillation. Its predictions provide a target for a smaller student to imitate, helping recover capabilities lost through pruning. Matching the teacher’s full prediction distribution transfers information about alternatives, not just the highest-probability answer. section 2.5 section 2.8

For each training batch, the teacher and student process the same input IDs and attention mask. The teacher’s forward pass runs without gradient calculation. KL divergence compares the student’s and teacher’s output distributions; backpropagation then updates the student’s weights. section 2.5

Listing 2.5 Step 4: Correcting the attempt
loss = F.kl_div(
   F.log_softmax(student_logits, dim=-1),
   F.softmax(teacher_logits, dim=-1),
   reduction='batchmean'
)
loss.backward()      #A
optimizer.step()     #B
optimizer.zero_grad()  #C
 

This loss step compares the student’s log probabilities with the teacher’s probabilities and updates the student. section 2.5

Configure the teacher in evaluation mode and freeze its parameters to avoid computing teacher gradients, saving GPU memory and time. Tokenize recovery data in batches and reserve separate validation data: evaluating perplexity on training samples can give an overly optimistic measure of recovery. section 6.1.2

Listing 6.2 Teacher model and tokenizer configuration
MODEL_NAME = "google/gemma-3-270m"
teacher_model = AutoModelForCausalLM.from_pretrained(
    MODEL_NAME,
    torch_dtype=torch.bfloat16 if torch.cuda.is_available() else torch.float32, #A
    device_map="auto" if torch.cuda.is_available() else None
)
 
teacher_model.eval() #B
for param in teacher_model.parameters():
    param.requires_grad = False #C
 
 
tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token
 

This configuration loads Gemma, freezes the teacher, and supplies a padding token when the tokenizer lacks one. section 6.1.2

Recovering a pruned model

An 18-layer gemma-3-270m model serves as teacher for a 16-layer student created by pruning a copy of it. The recovery experiment trains on 15,000 SlimPajama rows; pruning is relatively light, while recovery training needs substantially more compute and should run on a GPU. section 2.5

Replacing an agent call

A frontier model can serve as both teacher and evaluation bar for a specialized small model replacing one agent sub-task. Its existing outputs on the agent’s inputs define the behavior the replacement must learn and match. section 12.1

How Rearchitecting LLMs covers it

How deep it goes
Works through teacher configuration, shared-input logits, KL-divergence loss, student updates, and validation data preparation.
Examples
Uses an 18-layer Gemma teacher for a 16-layer student and a frontier model as teacher for an agent sub-task.
What you can do after
Applies knowledge distillation using PyTorch and OptiPFair to recover a pruned student from a teacher’s predictions.
How it connects
Connects knowledge distillation with depth pruning, prediction distributions, KL divergence, and perplexity.

Also explained in

Compare how 6 books explain teacher model →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.