liveBook · Manning

Student model

As explained in Rearchitecting LLMs

A student model is a model trained to reproduce a teacher model's behavior during knowledge distillation.

A student model learns to reproduce a teacher model’s behavior through knowledge distillation. After depth pruning, the student can recover lost capabilities while retaining the smaller model’s size and inference-speed benefits. section 2.5 section 2.8

The student learns to match the teacher’s prediction distribution, not just its final answer. Given the same input, both models produce logits; KL divergence measures their difference and supplies the loss used to update the student’s weights. The teacher does not need gradients. section 2.5 section 6.1.2

A recovery workflow has four stages:

  • Create the student by copying the teacher and pruning layers. OptiPFair’s prune_model can remove a specified number of final layers using DEPTH pruning. section 2.5
  • Prepare representative text, tokenize it into input IDs and attention masks, and batch the examples. A diverse dataset supports recovery of general capabilities rather than specialization in one text type. section 2.5 section 6.1.2
  • Run teacher inference without gradients, run the student on the same batch, and update student parameters using KL divergence and an optimizer. Training is more compute-intensive than structural pruning and should run on a GPU. section 2.5
  • Keep validation data separate from training data and compare the recovered student with the original and pruned models. Evaluating on training data can produce overly optimistic metrics. section 6.1.2 section 2.6
teacher_model = AutoModelForCausalLM.from_pretrained(
    MODEL_NAME,
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)
 
student_model = prune_model(  #A
    model=copy.deepcopy(teacher_model),  #A
    pruning_type="DEPTH",  #A
    num_layers_to_remove=LAYERS_TO_REMOVE,  #A
    layer_selection_method="last",  #A
    show_progress=True  #A
) 

The student starts as a deep copy of the teacher before final layers are removed. section 2.5

Listing 2.5 Step 4: Correcting the attempt
loss = F.kl_div(
   F.log_softmax(student_logits, dim=-1),
   F.softmax(teacher_logits, dim=-1),
   reduction='batchmean'
)
loss.backward()      #A
optimizer.step()     #B
optimizer.zero_grad()  #C
 

The loss compares student log probabilities with teacher probabilities, then updates only the student through its optimizer. section 2.5

Recovering a pruned Gemma model

An 18-layer gemma-3-270m teacher is copied and pruned to a 16-layer student, then trained on 15,000 SlimPajama rows. Compared with the original, the recovered student has 4.16% fewer parameters and takes 4.61 seconds rather than 5.12 seconds for inference. On the listed benchmarks, retained scores range from 87.2% to 102% of the original; recovery does not add back the removed parameters. section 2.5 section 2.6

Table 2.3 (Knowledge recovery analysis)
Metric / Model
gemma-3-270m
gemma-pruned
Student (Post-KD)
Retained capabilities
before /after KD
Parameters
268,098,176
256,950,912
256,950,912
-4.16
Inference Time
5.12s
4.61s
4.61s
10% Faster
arc_easy
0.59
0.49
0.54
83.1 / 91.5 %
winogrande
0.54
0.54
0.55
100 / 102 %
hellaswag
0.34
0.31
0.33
91.2 / 97.6 %
lambada_openai
0.43
0.36
0.38
83.7 / 87.2%

Compare the original, pruned, and recovered models on parameters, inference time, and controlled benchmark scores. section 2.6

How Rearchitecting LLMs covers it

How deep it goes
Walks through student construction, dataset preparation, KL-divergence training, and evaluation against original and pruned models.
Examples
Uses a pruned Gemma model trained on SlimPajama and a balanced mixture of Cosmopedia subsets for recovery.
What you can do after
Apply it via OptiPFair and PyTorch to construct, train, and evaluate a pruned student model.
How it connects
Connects knowledge distillation with depth pruning, teacher models, KL divergence, and validation.

Also explained in

Compare how 6 books explain student model →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.