Catastrophic forgetting

As explained in Reinforcement Learning from Human Feedback

Catastrophic forgetting is the rapid loss of previously learned capabilities or information when a model is trained on new objectives or data.

Catastrophic forgetting is the loss of prior capabilities during sequential training on new tasks. In post-training, supervised fine-tuning (SFT) can gain target-task performance while moving probability mass away from earlier capabilities; on-policy RL is presented as retaining more while achieving comparable or higher target-task gains. section 15.2.2

The proposed mechanism is the direction of KL divergence. SFT corresponds to forward KL, KL(π★ || πθ), and learns from target-distribution samples. RL corresponds to reverse KL, KL(πθ || π★), sampling from the current policy while optimizing reward with a reference-policy KL penalty. Forward KL must cover the target distribution and can pull mass from an old mode; reverse KL can move a new mode toward reward without disturbing the old one. section 15.2.2

Figure 15.1 Forgetting dynamics for forward KL (SFT) versus reverse KL (RL). The “old” mode represents prior knowledge, and the “new” mode represents the target task. Forward KL stretches the policy to cover the target and pulls mass away from the old mode (top right), whereas reverse KL shifts the new mode toward the target without disturbing the old mode (bottom right). From Chen et al. (2025), with permission of the author.

Compare the two distribution-shift patterns: forward KL stretches policy mass toward the target, whereas reverse KL shifts the new mode while preserving the old mode. section 15.2.2

A practical account of lower forgetting in online RL emphasizes policy drift:

• Among high-reward solutions, on-policy RL is biased toward policies closer in KL divergence to the initial policy. section 15.2.3
• Prior-task forgetting strongly correlates with KL distance between the trained and initial policies, even when that distance is measured on new-task inputs. section 15.2.3
• On-policy rather than offline sampling accounts for the reported generalization difference; negative gradients do not show a discernible additional effect. section 15.2.3

This makes KL drift from the base model a diagnostic for retention risk during specialization. Additional regularizers can preserve general language-modeling quality, including mixed pretraining gradients, negative log-likelihood on winning completions, reward-model margin losses, and reward-weighted DPO variants. section 15.2.3 section 15.4

Figure 15.2 Bias toward KL-minimal solutions reduces forgetting. (Left) Among policies that solve the new task, RL converges to those closest in KL to the base model. (Right) This KL bias yields higher prior-task retention at matched new-task performance compared to SFT. From Shenfeld, Pari, and Agrawal (2026). License CC-BY.

Inspect the trade-off at matched new-task performance: policies nearer the base model in KL retain more prior-task performance than SFT alternatives. section 15.2.3

How Reinforcement Learning from Human Feedback covers it

How deep it goes

Derives forward- and reverse-KL objectives, then links on-policy sampling and measured policy drift to retention.

Examples

Uses old-versus-new policy modes and matched-performance RL-versus-SFT comparisons to visualize retention.

What it assumes

Teaches KL directions using target and learned-policy notation before mapping them to SFT and RL.

What you can do after

Diagnose/debug retention risk by measuring KL drift from a base policy on specialized inputs.

How it connects

Connects supervised fine-tuning, reinforcement learning, KL divergence, on-policy sampling, RLHF, DPO, and regularization.

liveBookmanning.com

Read Reinforcement Learning from Human Feedback on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  The Art of AI Product DevelopmentCatastrophic forgetting is the loss of previously learned knowledge when a model is trained on new tasks or data.
• 

  liveBookmanning.com
  Generative AI in ActionCatastrophic forgetting is the loss of previously learned knowledge or capabilities when a model is trained on new data or tasks.

Compare how 3 books explain catastrophic forgetting →

Related concepts

• parameter-efficient fine-tuningaddressed by · Parameter-efficient fine-tuning adapts a pretrained model by training a limited subset of its parameters or added components rather than all its parameters.
• low-rank adaptationaddressed by · Low-rank adaptation is a parameter-efficient fine-tuning method that adapts a model by learning low-rank updates to its weights, typically while keeping most original weights fixed.
• Kullback–Leibler divergencemeasured by · Kullback–Leibler divergence is a nonnegative, directed measure of the information lost when one probability distribution approximates another, computed as an expected log ratio under the first distribution.
• learning rateaddressed by · A learning rate is a step-size parameter that controls the magnitude of updates made by an iterative learning or optimization process.

This explainer was generated by Manning's knowledge graph from the text of Reinforcement Learning from Human Feedback. The section links point to the parts of the book it draws on.