liveBook · Manning

Distillation

As explained in Applied Reinforcement Learning

A model-training technique in which a student model learns to reproduce useful outputs or behavior of a teacher model.

Distillation trains a smaller student model to mimic labels or signals produced by a more capable teacher, or oracle. The oracle runs offline on training data; the student then supplies the signal in deployment, where the oracle may be too slow, costly, or unavailable. In an RL system, this keeps the language model in the representation role: it produces state information, while the RL agent makes decisions. section 12.3 section 12.10

A practical distillation workflow is to:

  • Use the oracle to label curated training data offline. section 12.11 section 12.3
  • Fine-tune a smaller model to reproduce those labels, then deploy the student instead of the oracle. section 12.3
  • Measure how much of the oracle’s signal survives and whether it provides decision value. section 12.3

News signals for a portfolio manager

A frontier model such as DeepSeek reads news and generates reference market signals. It labels data offline; a smaller language model learns to reproduce those signals and supplies them to the portfolio manager’s state in deployment. The experiment measures whether useful signal survives the transfer. section 12.3

Figure 12.4 The distillation framing: an expensive oracle, a cheap deployable student.

The figure contrasts an expensive oracle that labels data offline with a cheaper student used in deployment. section 12.3

How Applied Reinforcement Learning covers it

How deep it goes
Explains offline labeling, student fine-tuning, deployment, and fidelity measurement without specifying a training algorithm.
Examples
Uses a frontier model to label news-derived market signals for a smaller model in a portfolio manager.
What it assumes
Assumes familiarity with an RL agent consuming state information and making decisions.
What you can do after
Apply it via offline oracle labeling, student fine-tuning, deployment, and measurement of surviving signal.
How it connects
Connects language-model fine-tuning for representation with reinforcement-learning agents that make sequential decisions.

Also explained in

Compare how 14 books explain distillation →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.