liveBook · Manning

Chapter 9: Managing model evolution, drift, and versioning

What you will learn in chapter 9 of LLM Customization and Fine-Tuning

This introduction explains why a deployed IT support model needs ongoing monitoring and maintenance even when its own weights have not changed.

After this chapter you will be able to

What it builds on

Section by section

Introduction

This introduction explains why a deployed IT support model needs ongoing monitoring and maintenance even when its own weights have not changed.

Key terms: model drift, domain specifics, data distributions, model lifecycle, monitoring, retrain, roll back

9.1 Model and data versioning

This section explains why model and data versioning is necessary to trace deployed answers, roll back degraded models, and maintain an audit trail.

Key terms: Reproducibility, checkpoint, training data, hyperparameters, Rollback, previous version, latency, error rates, uptime, Regulatory compliance

9.1.1 What to version

This subsection explains why model versions need registry metadata and a descriptive tag that makes versions easy to identify.

Key terms: model version entry, metadata, provenance, weights, versioning scheme, {name}-v{N}-{date}-{technique}-{base_model_short}-{data_vers, model versions, data version

9.1.2 Our approach: A single JSON file

This section presents a JSON-file model registry for small teams, explains when to move to a managed registry, and defines the registry’s operations and status lifecycle.

Key terms: single JSON file, version control, managed registry, multi-team access control, storage backend, register, list, promote, rollback, registered

9.1.3 Running the registry

This section shows how a JSON-backed model registry records deployments and why it must be kept current from the start.

Key terms: CLI, registry.json, promoting a model, active, retired, SFT, DPO, registry, versioning, checkpoint

9.2 Model drift: Understanding and responding

This section explains why an IT support model can lose accuracy in production and distinguishes four forms of drift.

Key terms: data drift, covariate shift, concept drift, prompt drift, model drift

9.2.1 Detection approaches

This subsection recommends combining two complementary drift-detection methods because no single detector covers every drift type.

Key terms: drift detector, embedding-based detector, canary prompts

9.2.2 Embedding-based drift detection

This section explains a lightweight way to detect changes in production questions by comparing their text vectors with a reference set.

Key terms: TF-IDF, shared vocabulary, bag of words, reference centroid, production centroid, cosine similarity, drift score, sentence-transformer embeddings, per-text vectorization, centroid-and-cosine machinery

9.2.3 Running the detector

Shows how to run and calibrate a prompt-drift detector, interpret its report, and use a canary check before deciding whether a red drift signal warrants retraining.

Key terms: reference JSONL, production JSONL, calibration baseline, drift score, alert threshold, top drift terms, minimum per-prompt similarity, training distribution, distribution drift, canary prompts

9.2.4 Canary prompts

This section explains how scheduled canary prompts reveal changes in model behavior, how to interpret them alongside input-distribution drift, and the trade-off of using an LLM as a drift detector.

Key terms: canary prompt, saved baseline, Token-F1, canary regression, distribution drift, canary regressions, leading indicator, confirming signal, alert thresholds, LLM-based drift detection

9.3 Handling failure and rollback

This section explains why model promotions need rollback capability in place before failures appear in production.

Key terms: drift detection, rollback capability, model promotion, evaluation suite, safety regression, response timeline

9.3.1 Rollback timeline

This section sets a rapid rollback target and explains how retained model versions and rehearsals make recovery practical.

Key terms: rollback, detection-to-verified-recovery, model versions, checkpoints, validated in production, non-production registry, real serving stack

9.3.2 Running the rollback demo

This section uses a fixed-metric simulation to show how a registry-backed threshold check triggers rollback and records the event.

Key terms: registry, rollback(), F1, safety_pass_rate, threshold check, structured rollback report, postmortem, timeline

9.3.3 Blue-green deployment for LLMs

This section explains how blue-green deployment makes LLM rollback a traffic-routing change and why a working rollback path must be ready before launch.

Key terms: blue-green deployment, blue environment, green environment, traffic, rollback, rollback path, checkpoint reload, deployment checklist, serving stack

9.4 Security and abuse monitoring

This section explains why security monitoring must continue after deployment, even for models that were previously tested and aligned.

Key terms: red-teamed, jailbroken, safety alignment, distilled student, DPO, adversarial behavior

9.4.1 The red-team prompt suite

This section explains how a red-team prompt suite checks four safety behaviors using category-specific pass thresholds and why its keyword-based results require recurring scrutiny.

Key terms: red-team prompt suite, keyword-based classifier, pass-rate thresholds, keyword matching, per-category thresholds

9.4.2 Running the safety monitor

This section shows how a safety monitor identifies deployment-blocking regressions, how a system-prompt guardrail affects the result, and why its keyword classifier needs safeguards.

Key terms: CLI, per-category pass-rate report, baseline, bias_fairness, alerts, safety-augmented system prompt, guardrail, retrain trigger, keyword classifier, expanded refusal vocabulary

9.4.3 When to retrain for safety

9.4.4 Production prompt monitoring

The section explains how to monitor production prompts and deployed models for emerging attacks, and why recurring red-team tests need to be complemented by multi-turn detection.

Key terms: production prompts, adversarial patterns, flagged prompts, safety_monitor.py, red-team suite, keyword-based classification, bias-fairness failure, multi-turn jailbreak detection, adversarial intent drift, runtime detectors

9.5 Base model evolution

This section frames base-model upgrades as strategic decisions and shows why a newer release may require retraining rather than a simple swap.

Key terms: base model, benchmarks, deprecated base, Qwen3-4B, Qwen3.5, adapter, retrain, revalidate

9.5.1 When to upgrade (green lights)

Identifies four signals that can justify upgrading a base model despite the cost of a full retraining cycle.

Key terms: base model, benchmark improvements, cost reduction, capability expansion, end-of-life, retraining cycle

9.5.2 When to stay (red lights)

This section identifies reasons to keep the current base model rather than risk an upgrade that may cause a regression.

Key terms: base model, regression, benchmark improvements, specific domain, full fine-tuning pipeline

9.5.3 The hidden cost of upgrading

This section explains why a base-model upgrade requires rebuilding and validating the adapted model pipeline, making upgrades a substantial planned effort.

Key terms: base model, weight space, LoRA adapter, SFT checkpoint, distilled student, DPO-optimized model, evaluation suite, LoRA/SFT training, distillation, DPO

9.5.4 Version locking

This section explains how to lock base-model versions for reproducible training and deployment, and how to handle upgrades deliberately.

Key terms: base model version, version pin, model registry, floating version tags, base-model upgrades, full-pipeline retraining

9.6 Long-term maintenance practices

This section explains why long-term model maintenance needs explicit ownership, a published review cadence, and tooling that catches problems early.

Key terms: weekly drift review, published cadence, named owners

9.6.1 Common pitfalls

This section identifies four operational failures that let production models degrade or become difficult to recover and explains the practices that prevent them.

Key terms: set-and-forget, cadence, drift alerts, canary prompts, rollback plan, previous model version, instant rollback, audit trail, model registry, retraining without validation

9.6.2 Maintenance cadence

This section explains why model maintenance needs a scheduled cadence with clear ownership, especially at weekly and annual intervals.

Key terms: maintenance cadence, cadence tiers, silent regressions

9.6.3 Monthly activities in detail

This section sets out a monthly review of canary results, user feedback, and safety tests to identify problems that need investigation or retraining.

Key terms: canary suite, baseline, Token-F1, negative feedback, failure categories, retrain signal, safety suite, red-team prompt suite, harmful-request, bias

9.6.4 Quarterly activities in detail

Quarterly maintenance is a senior-engineer review of data freshness, the need to retrain, and relevant base-model releases—not a schedule for automatic changes.

Key terms: data freshness, stale data, concept drift, drift detector's top terms, retraining assessment, metrics, degradation, base model review, base model provider, upgrade cycle

9.6.5 Annual activities

The section explains why an annual maintenance review should refresh the adaptation pipeline and reconsider whether newer techniques would improve the deployment.

Key terms: annual review, full pipeline refresh, technique evaluation

9.6.6 Managed platform options for monitoring and ops

This section explains how to choose managed tools for ongoing LLM operations and why a hybrid of cloud-native services and vendor-neutral observability often fits best.

Key terms: hybrid, vendor-neutral observability, portability, registry, access controls, CI/CD, artifacts, incident management, tabular ML, prompt drift

9.7 Retraining triggers and validation gates

This section distinguishes routine model checks from the outcome-based decision to retrain.

Key terms: cadence, input drift, output degradation, outcome-based

9.7.1 Retrain triggers

This subsection sets the evidence threshold for retraining and explains how to respond when input drift is detected.

Key terms: retrain trigger, input drift, leading indicator, canary prompts, output degradation, drift thresholds

9.7.2 Incremental retraining from checkpoint

This section explains how to retrain from a production checkpoint, validate the update, and limit the risk of forgetting earlier capabilities.

Key terms: most recent checkpoint, incremental retraining, delta, current production checkpoint, held-out set, register, promote, rollback, catastrophic forgetting, replay slice

9.7.3 Gating retrains

This section sets a deployment gate for retrained models and explains a research approach to preserving safety during fine-tuning.

Key terms: validation gate, overall eval suite, safety suite, category-level regressions, canary prompts, retrain on outcomes, checkpoint, full evaluation suite, safety-related gradients, task-utility gradients

9.8 Emerging base models and strategic planning

The section argues that reusable deployment foundations lower the cost of future models and that teams should plan across multiple horizons when deciding what to evaluate next.

Key terms: infrastructure, evaluation suites, data pipelines, deployment procedures, planning horizons, base models, techniques, platforms, evaluation budget

9.8.1 Planning horizons

This section lays out a phased plan for turning one monitored fine-tuned model into a repeatable, organization-wide model adaptation capability.

Key terms: 6-month plan, full lifecycle, rollback, retraining cycle, 1-year plan, reusable infrastructure, standardized training pipeline, shared evaluation framework, 3-year plan, organizational capability

9.8.2 Tracking the field

This section recommends selective monitoring of new base models, a quick evaluation screen before full evaluation, and delaying scale-out until after a successful deployment.

Key terms: canary prompts, eval suite, full evaluation cycle, base model landscape, sources, cadence, scale, successful deployment

9.9 End-to-end case study: One year of an IT support model

Introduces an illustrative IT support model deployment by showing how it is launched and monitored, and how monitoring, team decisions, and the registry work together.

Key terms: monitoring, registry, audit trail, it-support-v1, DPO-optimized model, canary and safety baselines, drift detection, cron

9.9.1 When to wait, when to act

An IT support model case study shows why a drift alert warrants monitoring when canary performance holds, but action when canary results and user feedback confirm a problem.

Key terms: drift detector, canary suite, Token-F1, two-signal discipline, updated dataset, SFT checkpoint, DPO, it-support-v2, deployment-gate evaluation

9.9.2 The rollback that proved the playbook

A production safety failure shows how monitoring, rollback, retraining, and a model registry support recovery and later audits.

Key terms: monthly safety run, bias_fairness, hard alert, model_registry rollback, biased examples, it-support-v3, full safety suite, model registry, retired, active

9.10 Closing the loop

This section brings the adaptation techniques together as an operational loop that keeps a domain-specific LLM useful and reliable over time.

Key terms: adaptation continuum, operational layer, feedback loop, drift detection, canary prompts, rollback procedure, feedback loops, playbook, retrain, rollback

9.11 Summary

9.12 References

This section lists references on model drift, monitoring, safety, and continual learning without explaining those topics.

Concepts explained in this chapter

← Chapter 8: Preference optimization: Teaching your model to judge

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.