liveBook · Manning

Rollback

As explained in LLM Customization and Fine-Tuning

Reverting a deployed system to an earlier version when the current version causes problems.

A newly deployed model can give incorrect or inconsistent answers while uptime, latency, and error dashboards remain green. Rollback restores an earlier version to limit the impact while the failure is investigated. It is the operational response to problems that monitoring or user reports reveal, including safety regressions and failures missed by evaluation. section 9.1 section 9.3

Rollback must be prepared before a promotion. Keep the previous two versions ready for instant serving, and retain old checkpoints until at least two newer versions have been validated in production. Two spare checkpoints for a full 4B model cost about 16 GB of disk; without them, recovery may require rebuilding the path back during an incident. section 9.3.1

A practical rollback playbook has four parts:

  • Detect degradation through monitoring or reports; infrastructure health alone may not reveal incorrect model behavior. section 9.3 section 9.1
  • Decide whether to revert when the new version causes a significant regression, including a safety failure. section 9.3 section 9.9.2
  • Execute the revert using a registry that identifies the active and prior versions and their checkpoint paths. section 9.1 section 9.9.2
  • Verify recovery, targeting less than one hour from detection to verified rollback. Practice first in a non-production registry, then quarterly against the real serving stack. section 9.3.1
Figure 9.4 Rollback timeline: detection, decision, execution, and verification inside one hour.

The timeline organizes detection, decision, execution, and verification around the one-hour recovery target. section 9.3.1

Safety regression after retraining

In an illustrative help-desk deployment, a monthly safety run flags a bias-fairness pass rate of 0.50 against a 1.00 threshold. A registry rollback restores v1 and retires v2, with verified recovery in under an hour. The team removes biased training examples, retrains as v3, and runs the full safety suite before promotion; the failed v2 remains in the registry for audit. section 9.9.2 section 9.0

A registry also supports the investigation after recovery: version tags, training technique, base model, data hash, checkpoint path, and evaluation metrics provide an audit trail. Rollback is not a substitute for validating the corrected model before its next deployment. section 9.11 section 9.6.1

How LLM Customization and Fine-Tuning covers it

How deep it goes
Sets a recovery target and retention policy, outlines response phases, and recommends quarterly drills against the serving stack.
Examples
Uses an illustrative help-desk safety regression and a registry rollback from v2 to v1, followed by a validated retrain.
What it assumes
Requires retained checkpoints, a way to identify the prior active version, and a serving stack capable of restoring it.
What you can do after
Practice rollback using a non-production registry and quarterly drills against the real serving stack.
How it connects
Connects rollback to model versioning, drift detection, safety monitoring, and evaluation.

Also explained in

Compare how 14 books explain rollback →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.