Chapter 9: Managing model evolution, drift, and versioning
This introduction explains why a deployed IT support model needs ongoing monitoring and maintenance even when its own weights have not changed.
After this chapter you will be able to
- Drift in an unchanged model. Explain how an unchanged IT support model can produce worse results when its production environment changes, and describe the domain-specific signs of that decline.
- Ongoing production model lifecycle. Describe why deployment begins ongoing work and trace the high-level lifecycle from training through production monitoring to retraining or rollback.
- Deployed Model Provenance. Explain how versioning lets a team trace a deployed answer to the exact checkpoint, training data, and hyperparameters that produced it.
- Fast Model Rollback. Explain why rollback depends on knowing the previously active version and where its checkpoint is stored, even when infrastructure metrics look healthy.
- Model version metadata purposes. Explain that each model version entry needs enough metadata to reproduce its training run, locate its weights, and demonstrate provenance to an auditor.
- Descriptive model version tags. Describe the version tag pattern and explain why it is deliberately verbose: it lets someone distinguish versions by their number, date, training technique, base model, and data version at a glance.
- JSON registry trade-off. Justify using a single JSON file instead of an enterprise MLOps registry when a team is getting started.
- Registry operations and statuses. Describe the registry’s four operations, its model-status progression, and what rollback reverses.
- Registry and deployment coordination. Describe how the registry CLI reads or updates the JSON file, and explain why promoting a model and updating its registry record should happen in one deployment step.
- Predeployment versioning value. Justify versioning even the first model before deployment, including its value for rollback and later audit requests.
- Four forms of model drift. Distinguish data, concept, prompt, and model drift by what changes in each case, using the IT support examples.
- Complementary drift detection. Explain why the pragmatic approach combines a continuous embedding-based detector with periodic canary prompts rather than relying on one detector.
- TF-IDF Text Representation. Explain how the detector uses TF-IDF to turn reference and production texts into vectors that emphasize distinctive vocabulary.
- Centroid Drift Score. Trace how the detector reduces the reference and production sets to centroids and computes a drift score from their cosine similarity.
- Same-domain drift baseline. Describe how to establish a calibration baseline by comparing training prompts with same-domain validation prompts, then set alert thresholds relative to that baseline rather than relying on defaults.
- Shifted-topic drift evidence. Distinguish ordinary vocabulary-level drift from an actionable shift by using both the drift score and the top drift terms; describe what low per-prompt similarity adds.
- Drift and canary signals. Explain why a red distribution-drift signal should trigger a behavioral canary check before a retraining decision, while a low drift score calls for monitoring.
- Canary prompt behavior checks. Explain how fixed prompts test model behavior over time and how Token-F1 comparisons with saved baselines flag regressions.
- Complementary drift monitoring signals. Distinguish input-distribution monitoring from canary testing and describe how to use their signals together without retraining prematurely.
- Detection Versus Rollback. Distinguish detecting a model change from reverting a problematic version, and explain why rollback capability must be built into deployment before it is needed.
- Rapid rollback target. Explain why rollback speed matters and state the target of verified recovery within one hour of detection.
- Ready previous model versions. Describe which prior model versions to retain and why they must remain ready to serve.
- Registry-backed rollback workflow. Trace how the demo registers and promotes v1, registers and promotes v2, then checks v2’s degraded metrics and calls rollback() when they fail the thresholds. Explain that this is a workflow simulation, not a run with loaded models.
- Blue-green traffic switching. Trace how traffic moves between the current production model and a candidate model, including how to roll back if the candidate fails. Explain why this is useful for LLMs and what it costs.
- Ongoing model safety monitoring. Explain why deployment-time red-teaming is not enough: new jailbreak techniques can emerge, and adaptation can change a model’s safety behavior. Describe why distilled students need independent safety re-testing and why adversarial behavior must be watched for continuously.
- Red-team monitor checks. Describe how the prompt suite uses a keyword-based classifier to count responses and applies pass-rate thresholds across four safety categories.
- Safety report and deployment decision. Trace how the CLI tests safety prompts, reports category pass rates, and flags failures; explain why the SFT model's reported result blocks deployment.
- System prompt guardrail limits. Explain what changed when the same model was rerun with a safety-augmented system prompt and why that result is not a permanent fix.
- Production prompt review. Describe how to sample and review production prompts for adversarial patterns while respecting privacy requirements.
- Recurring deployed-model safety checks. Explain how to repeat and update red-team checks against the deployed model, and when a result requires investigation.
- Base Model Upgrade Tradeoffs. Explain why stronger benchmarks alone do not settle whether to upgrade a deployed model. Upgrading too aggressively and remaining on a deprecated base can both cost months of effort.
- Adapter Compatibility Across Releases. Describe why the Qwen3.5 release is not a drop-in replacement for the book's Qwen3-4B base: an adapter trained on Qwen3-4B does not load on Qwen3.5, so upgrading requires retraining and revalidation.
- Base Model Upgrade Signals. Describe the four green lights for a base model upgrade: better performance on domain-relevant benchmarks, lower cost or expanded capabilities, provider deprecation of the current model, and an already-planned retraining cycle.
- Current Model and Domain Fit. Explain why a well-performing current model and general-task benchmark gains are not, by themselves, reasons to switch base models.
- Base version dependencies. Explain why an existing LoRA adapter or other adapted model cannot simply be carried over to a new base-model version.
- Upgrade retraining sequence. Trace the training work required after selecting a new base model, including the conditional steps for distillation and DPO.
- Exact base-model version locking. Explain why training scripts and deployment configuration should pin the same exact base-model version rather than use a floating tag, and why that version should be recorded in the model registry.
- Sustained maintenance ownership. Explain how production model regressions can accumulate when routine reviews have no owner, and describe the maintenance process proposed to prevent that.
- Ongoing monitoring discipline. Explain why deploying a model and then ceasing to monitor it is risky, and describe the basic maintenance discipline the section recommends.
- Ready rollback path. Explain why a team should retain the previous model version when deploying a retrained one.
- Validation before retrained deployment. Explain why detecting drift and retraining do not, by themselves, justify deploying a new model.
- Maintenance cadence and ownership. Explain why maintenance is organized into four cadence tiers, with an activity, owner, and tool assigned to each interval, and describe the risks of skipping weekly or annual maintenance.
- Monthly canary prompt review. Describe how to compare monthly canary results with the deployment baseline and when to investigate a decline.
- User feedback failure trends. Explain how categorized negative feedback can reveal a need to retrain that automated metrics may miss.
- Monthly safety regression checks. Describe the monthly safety-suite comparison and the response to a new harmful-request or bias failure.
- Training data freshness review. Describe how to check whether training data still reflects current products, policies, processes, and terminology, and why stale data can cause concept drift.
- Evidence based retraining assessment. Explain why the quarterly review assesses the need for retraining rather than initiating retraining on a fixed schedule.
- Annual model maintenance review. Explain why an annual review is needed even when monthly and quarterly checks look healthy, and describe its two activities: refreshing the pipeline with updated data and evaluating newer approaches.
- Hybrid operations strategy. Justify using an existing cloud for registry, serving, and rollback while adding vendor-neutral tools for LLM-specific drift and quality monitoring.
- Outcome Based Retraining Decisions. Explain why a drift alert or scheduled review should prompt assessment rather than automatic retraining, and why the decision should depend on whether model performance has degraded.
- Evidence Based Retraining Decisions. Explain why input drift alone does not justify retraining, and describe how to investigate a drift alert before deciding whether to retrain.
- Checkpoint retraining advantage. Explain why retraining starts from the most recent checkpoint instead of the base model.
- Incremental retraining and release loop. Trace how refreshed data becomes a candidate model and how its evaluation determines the next step.
- Catastrophic forgetting safeguards. Explain the main risk of repeated incremental retraining and the two habits recommended to reduce it.
- Retrain validation gate checks. State the checks a retrained model must pass before deployment: overall evaluation against production, safety thresholds, category-level regression limits, and canary prompt behavior.
- Retrain decisions and failed gates. Explain when to retrain and what to do if a retrained model fails validation. Describe the section’s instruction to start from a checkpoint and run the full evaluation suite before deployment.
- Reusable deployment foundations. Explain why a team’s later model deployments can be faster than its first and why treating each model as a one-off loses that advantage.
- Long-horizon evaluation planning. Describe what planning beyond a single model involves and why a team tracks developments in base models, techniques, and platforms.
- First Production Lifecycle. Explain why the six-month priority is to run one fine-tuned model through its full production lifecycle before trying to scale.
- Reusable Scaling Infrastructure. Describe how the one-year plan turns a successful deployment into a repeatable process for additional use cases.
- Rapid base model screening. Explain how to decide whether a newly released base model warrants a full evaluation cycle.
- Monitoring decisions and audit trail. Explain the operational pattern illustrated by the deployment: monitoring surfaces a signal, the team decides whether to act, and the registry preserves an audit trail.
- Corroborating drift before retraining. Explain why the team waited after the month 4 drift alert but acted in month 7. Distinguish shifted inputs from evidence that the model’s answers have deteriorated.
- Safety alert and rollback. Trace how the safety alert led the team to investigate and roll back it-support-v2 to v1.
- Cleaned retrain validation. Describe what the team changed before promoting the replacement model.
- Operational feedback across adaptation. Explain that the operational layer wraps every rung of the adaptation continuum: drift detection, canary prompts, and rollback provide feedback whether a team uses a few-shot prompt template or a DPO checkpoint.
What it builds on
- Lightweight operationalization (chapter 2) · The earlier introduction to operationalizing an adapted model.
- Data lineage (chapter 3) · Data lineage.
- Data versioning and experiment tracking (chapter 3) · Earlier discussion of data versioning and experiment tracking.
- Does your training data reflect your production distribution? (chapter 3) · The distinction between the training-data distribution and the questions seen in production.
- Why safety does not transfer (chapter 7) · Safety alignment does not transfer through distillation.
- Dual-objective optimization (chapter 8) · Preference optimization can affect safety alongside helpfulness.
- Safety regression testing in practice (chapter 5) · The earlier safety release gate provides a coarser comparison with the base model.
- Student model robustness checks (chapter 7) · Independent safety testing of distilled students.
Section by section
Introduction
This introduction explains why a deployed IT support model needs ongoing monitoring and maintenance even when its own weights have not changed.
- Drift in an unchanged model: Explain how an unchanged IT support model can produce worse results when its production environment changes, and describe the domain-specific signs of that decline.
Why it matters: A stable model version does not guarantee stable production behavior. - Ongoing production model lifecycle: Describe why deployment begins ongoing work and trace the high-level lifecycle from training through production monitoring to retraining or rollback.
Why it matters: Treating deployment as the finish line leaves production changes without a response.
Key terms: model drift, domain specifics, data distributions, model lifecycle, monitoring, retrain, roll back
9.1 Model and data versioning
This section explains why model and data versioning is necessary to trace deployed answers, roll back degraded models, and maintain an audit trail.
- Deployed Model Provenance: Explain how versioning lets a team trace a deployed answer to the exact checkpoint, training data, and hyperparameters that produced it.
Why it matters: That trace lets the team investigate why a model gave a particular answer. - Fast Model Rollback: Explain why rollback depends on knowing the previously active version and where its checkpoint is stored, even when infrastructure metrics look healthy.
Why it matters: A team cannot quickly restore a working model if it has to search for the version and weights during an incident. - Model Decision Audit Trail: Describe how a versioned registry supports audits by recording which model made which decisions and when.
Why it matters: The trail helps an organization account for model decisions during an audit.
Key terms: Reproducibility, checkpoint, training data, hyperparameters, Rollback, previous version, latency, error rates, uptime, Regulatory compliance
9.1.1 What to version
This subsection explains why model versions need registry metadata and a descriptive tag that makes versions easy to identify.
- Model version metadata purposes: Explain that each model version entry needs enough metadata to reproduce its training run, locate its weights, and demonstrate provenance to an auditor.
Why it matters: A version identifier alone is not enough to reproduce or audit a model. - Descriptive model version tags: Describe the version tag pattern and explain why it is deliberately verbose: it lets someone distinguish versions by their number, date, training technique, base model, and data version at a glance.
Why it matters: A descriptive tag helps people pick out the intended model when several versions coexist.
Key terms: model version entry, metadata, provenance, weights, versioning scheme, {name}-v{N}-{date}-{technique}-{base_model_short}-{data_vers, model versions, data version
9.1.2 Our approach: A single JSON file
This section presents a JSON-file model registry for small teams, explains when to move to a managed registry, and defines the registry’s operations and status lifecycle.
- JSON registry trade-off: Justify using a single JSON file instead of an enterprise MLOps registry when a team is getting started.
Why it matters: The registry can be useful without requiring a platform before the team needs one. - Managed registry transition: State when the section recommends migrating from the JSON file to a managed registry and what remains unchanged.
Why it matters: The simple registry is a starting point, not a commitment to keep using a file at every scale. - Registry operations and statuses: Describe the registry’s four operations, its model-status progression, and what rollback reverses.
Why it matters: These operations and statuses give the team a shared way to track which model version is in production.
Key terms: single JSON file, version control, managed registry, multi-team access control, storage backend, register, list, promote, rollback, registered
9.1.3 Running the registry
This section shows how a JSON-backed model registry records deployments and why it must be kept current from the start.
- Registry and deployment coordination: Describe how the registry CLI reads or updates the JSON file, and explain why promoting a model and updating its registry record should happen in one deployment step.
Why it matters: A deployment record is useful only if it stays in step with the model that was promoted. - Readable model version history: Explain how the JSON registry preserves the status and evaluation history of model versions, using the active SFT model and retired DPO experiment as examples.
Why it matters: A retained history lets people and scripts see which version is active and what happened to earlier candidates. - Predeployment versioning value: Justify versioning even the first model before deployment, including its value for rollback and later audit requests.
Why it matters: Versioning before an incident preserves the information needed to recover and account for deployments.
Key terms: CLI, registry.json, promoting a model, active, retired, SFT, DPO, registry, versioning, checkpoint
9.2 Model drift: Understanding and responding
This section explains why an IT support model can lose accuracy in production and distinguishes four forms of drift.
- Four forms of model drift: Distinguish data, concept, prompt, and model drift by what changes in each case, using the IT support examples.
Why it matters: Identifying what changed helps you interpret an accuracy drop without assuming the deployed model itself changed.
Key terms: data drift, covariate shift, concept drift, prompt drift, model drift
9.2.1 Detection approaches
This subsection recommends combining two complementary drift-detection methods because no single detector covers every drift type.
- Complementary drift detection: Explain why the pragmatic approach combines a continuous embedding-based detector with periodic canary prompts rather than relying on one detector.
Why it matters: Using complementary checks helps avoid relying on one detector to catch every kind of drift.
Key terms: drift detector, embedding-based detector, canary prompts
9.2.2 Embedding-based drift detection
This section explains a lightweight way to detect changes in production questions by comparing their text vectors with a reference set.
- TF-IDF Text Representation: Explain how the detector uses TF-IDF to turn reference and production texts into vectors that emphasize distinctive vocabulary.
Why it matters: The detector needs comparable numeric representations before it can spot a shift in questions. - Centroid Drift Score: Trace how the detector reduces the reference and production sets to centroids and computes a drift score from their cosine similarity.
Why it matters: The score summarizes how much recent production questions differ from the reference questions. - Richer Vectorizer Substitution: Describe what changes, and what stays the same, when sentence-transformer embeddings replace TF-IDF vectors.
Why it matters: A production system can use richer text representations without changing the basic drift calculation.
Key terms: TF-IDF, shared vocabulary, bag of words, reference centroid, production centroid, cosine similarity, drift score, sentence-transformer embeddings, per-text vectorization, centroid-and-cosine machinery
9.2.3 Running the detector
Shows how to run and calibrate a prompt-drift detector, interpret its report, and use a canary check before deciding whether a red drift signal warrants retraining.
- Same-domain drift baseline: Describe how to establish a calibration baseline by comparing training prompts with same-domain validation prompts, then set alert thresholds relative to that baseline rather than relying on defaults.
Why it matters: A calibrated baseline helps keep ordinary same-domain variation from becoming a false alarm. - Shifted-topic drift evidence: Distinguish ordinary vocabulary-level drift from an actionable shift by using both the drift score and the top drift terms; describe what low per-prompt similarity adds.
Why it matters: The terms and individual outliers help explain what changed instead of leaving you with only an alert number. - Drift and canary signals: Explain why a red distribution-drift signal should trigger a behavioral canary check before a retraining decision, while a low drift score calls for monitoring.
Why it matters: Changed prompts do not by themselves prove that answers have degraded; the canary checks behavior before action.
Key terms: reference JSONL, production JSONL, calibration baseline, drift score, alert threshold, top drift terms, minimum per-prompt similarity, training distribution, distribution drift, canary prompts
9.2.4 Canary prompts
This section explains how scheduled canary prompts reveal changes in model behavior, how to interpret them alongside input-distribution drift, and the trade-off of using an LLM as a drift detector.
- Canary prompt behavior checks: Explain how fixed prompts test model behavior over time and how Token-F1 comparisons with saved baselines flag regressions.
Why it matters: A repeatable behavior check can reveal a regression before it becomes a broader production problem. - Complementary drift monitoring signals: Distinguish input-distribution monitoring from canary testing and describe how to use their signals together without retraining prematurely.
Why it matters: Using both signals helps catch real behavioral failures without treating every input shift as a reason to retrain. - LLM drift detector trade-off: Describe the proposed use of an LLM to classify whether new inputs fit an expected distribution, and explain its monitoring risk.
Why it matters: A convenient monitoring method is less trustworthy if its own behavior can drift unnoticed.
Key terms: canary prompt, saved baseline, Token-F1, canary regression, distribution drift, canary regressions, leading indicator, confirming signal, alert thresholds, LLM-based drift detection
9.3 Handling failure and rollback
This section explains why model promotions need rollback capability in place before failures appear in production.
- Detection Versus Rollback: Distinguish detecting a model change from reverting a problematic version, and explain why rollback capability must be built into deployment before it is needed.
Why it matters: Spotting a failure is not enough if reverting the model takes too long. - Model Promotion Risk: Explain why a promoted model can fail despite pre-release checks and why the response timeline matters.
Why it matters: Pre-release checks reduce risk but cannot eliminate the need to respond to production failures.
Key terms: drift detection, rollback capability, model promotion, evaluation suite, safety regression, response timeline
9.3.1 Rollback timeline
This section sets a rapid rollback target and explains how retained model versions and rehearsals make recovery practical.
- Rapid rollback target: Explain why rollback speed matters and state the target of verified recovery within one hour of detection.
Why it matters: A slow rollback leaves users exposed to bad answers even when no outage is reported. - Ready previous model versions: Describe which prior model versions to retain and why they must remain ready to serve.
Why it matters: A rollback cannot be quick if the model you need is no longer ready to serve. - Rollback rehearsal cadence: Describe where to practice rollback initially and how often to rehearse it on the real serving stack.
Why it matters: Rehearsal avoids making an urgent incident the team’s first end-to-end rollback attempt.
Key terms: rollback, detection-to-verified-recovery, model versions, checkpoints, validated in production, non-production registry, real serving stack
9.3.2 Running the rollback demo
This section uses a fixed-metric simulation to show how a registry-backed threshold check triggers rollback and records the event.
- Registry-backed rollback workflow: Trace how the demo registers and promotes v1, registers and promotes v2, then checks v2’s degraded metrics and calls rollback() when they fail the thresholds. Explain that this is a workflow simulation, not a run with loaded models.
Why it matters: The demo isolates the connection between deployment state, monitoring thresholds, and rollback without requiring real model inference. - Structured rollback report: Describe the report the script writes and its intended use in a postmortem.
Why it matters: A recorded rollback timeline helps explain the incident afterward.
Key terms: registry, rollback(), F1, safety_pass_rate, threshold check, structured rollback report, postmortem, timeline
9.3.3 Blue-green deployment for LLMs
This section explains how blue-green deployment makes LLM rollback a traffic-routing change and why a working rollback path must be ready before launch.
- Blue-green traffic switching: Trace how traffic moves between the current production model and a candidate model, including how to roll back if the candidate fails. Explain why this is useful for LLMs and what it costs.
Why it matters: Keeping the previous model ready to serve can make recovery much faster when a candidate fails. - Prelaunch rollback readiness: State when a rollback path must be in place and how the demonstrated approach relates to a real deployment.
Why it matters: A simulated rollback is not a recovery plan until it works with the system that will serve the model.
Key terms: blue-green deployment, blue environment, green environment, traffic, rollback, rollback path, checkpoint reload, deployment checklist, serving stack
9.4 Security and abuse monitoring
This section explains why security monitoring must continue after deployment, even for models that were previously tested and aligned.
- Ongoing model safety monitoring: Explain why deployment-time red-teaming is not enough: new jailbreak techniques can emerge, and adaptation can change a model’s safety behavior. Describe why distilled students need independent safety re-testing and why adversarial behavior must be watched for continuously.
Why it matters: Passing safety tests at deployment does not guarantee that a model remains safe in production.
Key terms: red-teamed, jailbroken, safety alignment, distilled student, DPO, adversarial behavior
9.4.1 The red-team prompt suite
This section explains how a red-team prompt suite checks four safety behaviors using category-specific pass thresholds and why its keyword-based results require recurring scrutiny.
- Red-team monitor checks: Describe how the prompt suite uses a keyword-based classifier to count responses and applies pass-rate thresholds across four safety categories.
Why it matters: These checks make it possible to spot category-specific safety regressions after deployment. - Limits of keyword monitoring: Explain why the suite uses different thresholds and is rerun monthly rather than treated as a one-time deployment check.
Why it matters: A passing keyword check is useful, but it is not proof that every subtle safety failure has been caught.
Key terms: red-team prompt suite, keyword-based classifier, pass-rate thresholds, keyword matching, per-category thresholds
9.4.2 Running the safety monitor
This section shows how a safety monitor identifies deployment-blocking regressions, how a system-prompt guardrail affects the result, and why its keyword classifier needs safeguards.
- Safety report and deployment decision: Trace how the CLI tests safety prompts, reports category pass rates, and flags failures; explain why the SFT model's reported result blocks deployment.
Why it matters: An overall score can conceal a category failure serious enough to stop a release. - System prompt guardrail limits: Explain what changed when the same model was rerun with a safety-augmented system prompt and why that result is not a permanent fix.
Why it matters: A passing rerun can mitigate an immediate failure without proving the underlying model is fixed. - Keyword classifier failure modes: Explain how a keyword-based safety classifier can both miss valid refusals and wrongly credit an unsafe answer, and describe the safeguards used here.
Why it matters: A misleading classifier can make a safety report look worse or better than the responses warrant.
Key terms: CLI, per-category pass-rate report, baseline, bias_fairness, alerts, safety-augmented system prompt, guardrail, retrain trigger, keyword classifier, expanded refusal vocabulary
9.4.3 When to retrain for safety
9.4.4 Production prompt monitoring
The section explains how to monitor production prompts and deployed models for emerging attacks, and why recurring red-team tests need to be complemented by multi-turn detection.
- Production prompt review: Describe how to sample and review production prompts for adversarial patterns while respecting privacy requirements.
Why it matters: Reviewing real prompts can reveal adversarial behavior that a fixed test suite has not anticipated. - Recurring deployed-model safety checks: Explain how to repeat and update red-team checks against the deployed model, and when a result requires investigation.
Why it matters: Recurring checks can catch safety regressions after deployment, while updates keep the tests relevant to emerging attacks. - Multi-turn attack detection: Explain why single-turn red-team tests are insufficient for a multi-turn deployment and describe the runtime detection approach presented here.
Why it matters: A conversation can become adversarial gradually even when individual turns do not match a known test prompt.
Key terms: production prompts, adversarial patterns, flagged prompts, safety_monitor.py, red-team suite, keyword-based classification, bias-fairness failure, multi-turn jailbreak detection, adversarial intent drift, runtime detectors
9.5 Base model evolution
This section frames base-model upgrades as strategic decisions and shows why a newer release may require retraining rather than a simple swap.
- Base Model Upgrade Tradeoffs: Explain why stronger benchmarks alone do not settle whether to upgrade a deployed model. Upgrading too aggressively and remaining on a deprecated base can both cost months of effort.
Why it matters: It helps you treat a promising release as a decision to evaluate, not an automatic upgrade. - Adapter Compatibility Across Releases: Describe why the Qwen3.5 release is not a drop-in replacement for the book's Qwen3-4B base: an adapter trained on Qwen3-4B does not load on Qwen3.5, so upgrading requires retraining and revalidation.
Why it matters: An incompatible adapter changes the work involved in adopting a new base model.
Key terms: base model, benchmarks, deprecated base, Qwen3-4B, Qwen3.5, adapter, retrain, revalidate
9.5.1 When to upgrade (green lights)
Identifies four signals that can justify upgrading a base model despite the cost of a full retraining cycle.
- Base Model Upgrade Signals: Describe the four green lights for a base model upgrade: better performance on domain-relevant benchmarks, lower cost or expanded capabilities, provider deprecation of the current model, and an already-planned retraining cycle.
Why it matters: These signals help you decide when an upgrade is worth the retraining effort.
Key terms: base model, benchmark improvements, cost reduction, capability expansion, end-of-life, retraining cycle
9.5.2 When to stay (red lights)
This section identifies reasons to keep the current base model rather than risk an upgrade that may cause a regression.
- Current Model and Domain Fit: Explain why a well-performing current model and general-task benchmark gains are not, by themselves, reasons to switch base models.
Why it matters: An upgrade is not useful if it makes the application your stakeholders rely on worse. - Upgrade Maturity and Capacity: Describe how a new base model’s age and the team’s fine-tuning capacity affect the decision to upgrade.
Why it matters: Waiting for issues to surface and accounting for retraining capacity can prevent a premature upgrade.
Key terms: base model, regression, benchmark improvements, specific domain, full fine-tuning pipeline
9.5.3 The hidden cost of upgrading
This section explains why a base-model upgrade requires rebuilding and validating the adapted model pipeline, making upgrades a substantial planned effort.
- Base version dependencies: Explain why an existing LoRA adapter or other adapted model cannot simply be carried over to a new base-model version.
Why it matters: A new base model is not a drop-in replacement for the versions of your adapted models already in production. - Upgrade retraining sequence: Trace the training work required after selecting a new base model, including the conditional steps for distillation and DPO.
Why it matters: Upgrading the base means repeating the adaptation work, not just downloading a replacement model. - Upgrade validation and cadence: Describe the checks and deployment preparation that follow retraining, and state the time and frequency the section recommends planning for.
Why it matters: The validation and rollback work makes an upgrade a scheduled release decision rather than a routine model swap.
Key terms: base model, weight space, LoRA adapter, SFT checkpoint, distilled student, DPO-optimized model, evaluation suite, LoRA/SFT training, distillation, DPO
9.5.4 Version locking
This section explains how to lock base-model versions for reproducible training and deployment, and how to handle upgrades deliberately.
- Exact base-model version locking: Explain why training scripts and deployment configuration should pin the same exact base-model version rather than use a floating tag, and why that version should be recorded in the model registry.
Why it matters: Exact pins and registry records make it possible to identify the base model behind a deployed or retrained system. - Controlled base-model upgrades: Describe how to approach base-model upgrades instead of treating a new base model as a drop-in replacement.
Why it matters: An upgrade is a deliberate retraining decision, not an automatic model swap.
Key terms: base model version, version pin, model registry, floating version tags, base-model upgrades, full-pipeline retraining
9.6 Long-term maintenance practices
This section explains why long-term model maintenance needs explicit ownership, a published review cadence, and tooling that catches problems early.
- Sustained maintenance ownership: Explain how production model regressions can accumulate when routine reviews have no owner, and describe the maintenance process proposed to prevent that.
Why it matters: A model can degrade even without a faulty release if routine checks stop happening.
Key terms: weekly drift review, published cadence, named owners
9.6.1 Common pitfalls
This section identifies four operational failures that let production models degrade or become difficult to recover and explains the practices that prevent them.
- Ongoing monitoring discipline: Explain why deploying a model and then ceasing to monitor it is risky, and describe the basic maintenance discipline the section recommends.
Why it matters: Regular attention helps catch deterioration while the team can still address it. - Ready rollback path: Explain why a team should retain the previous model version when deploying a retrained one.
Why it matters: A retained version lets the team respond quickly if a replacement causes problems. - Consistent model audit trail: Describe the question an audit trail must answer and why consistent use of the model registry matters.
Why it matters: A consistent record lets the team answer questions about past outputs promptly. - Validation before retrained deployment: Explain why detecting drift and retraining do not, by themselves, justify deploying a new model.
Why it matters: Validation can catch regressions that the signal prompting a retrain would miss.
Key terms: set-and-forget, cadence, drift alerts, canary prompts, rollback plan, previous model version, instant rollback, audit trail, model registry, retraining without validation
9.6.2 Maintenance cadence
This section explains why model maintenance needs a scheduled cadence with clear ownership, especially at weekly and annual intervals.
- Maintenance cadence and ownership: Explain why maintenance is organized into four cadence tiers, with an activity, owner, and tool assigned to each interval, and describe the risks of skipping weekly or annual maintenance.
Why it matters: A model can appear healthy while missed maintenance creates regressions or undermines future retraining.
Key terms: maintenance cadence, cadence tiers, silent regressions
9.6.3 Monthly activities in detail
This section sets out a monthly review of canary results, user feedback, and safety tests to identify problems that need investigation or retraining.
- Monthly canary prompt review: Describe how to compare monthly canary results with the deployment baseline and when to investigate a decline.
Why it matters: A regular baseline comparison helps reveal performance changes before they go unexplained. - User feedback failure trends: Explain how categorized negative feedback can reveal a need to retrain that automated metrics may miss.
Why it matters: Feedback can expose new kinds of failure that routine metrics do not capture. - Monthly safety regression checks: Describe the monthly safety-suite comparison and the response to a new harmful-request or bias failure.
Why it matters: A model that passed safety tests at deployment can develop or reveal failures that call for retraining.
Key terms: canary suite, baseline, Token-F1, negative feedback, failure categories, retrain signal, safety suite, red-team prompt suite, harmful-request, bias
9.6.4 Quarterly activities in detail
Quarterly maintenance is a senior-engineer review of data freshness, the need to retrain, and relevant base-model releases—not a schedule for automatic changes.
- Training data freshness review: Describe how to check whether training data still reflects current products, policies, processes, and terminology, and why stale data can cause concept drift.
Why it matters: A freshness check helps reveal when the model’s training examples no longer match the questions it receives. - Evidence based retraining assessment: Explain why the quarterly review assesses the need for retraining rather than initiating retraining on a fixed schedule.
Why it matters: Reviewing evidence before retraining avoids treating a consequential model change as routine maintenance. - Domain relevant base model review: Describe how to review a new base-model release during quarterly maintenance.
Why it matters: A release is useful to track only if its improvements matter to the model’s use case.
Key terms: data freshness, stale data, concept drift, drift detector's top terms, retraining assessment, metrics, degradation, base model review, base model provider, upgrade cycle
9.6.5 Annual activities
The section explains why an annual maintenance review should refresh the adaptation pipeline and reconsider whether newer techniques would improve the deployment.
- Annual model maintenance review: Explain why an annual review is needed even when monthly and quarterly checks look healthy, and describe its two activities: refreshing the pipeline with updated data and evaluating newer approaches.
Why it matters: A model can pass routine checks yet slowly become less effective or fall behind better available approaches.
Key terms: annual review, full pipeline refresh, technique evaluation
9.6.6 Managed platform options for monitoring and ops
This section explains how to choose managed tools for ongoing LLM operations and why a hybrid of cloud-native services and vendor-neutral observability often fits best.
- Hybrid operations strategy: Justify using an existing cloud for registry, serving, and rollback while adding vendor-neutral tools for LLM-specific drift and quality monitoring.
Why it matters: It gives you a practical default without requiring one vendor to be best at every operational task. - Registry evaluation criteria: Explain why registry selection should focus on access controls and operational integrations rather than basic field tracking.
Why it matters: These differences determine whether a registry fits your deployment and compliance processes. - LLM drift monitoring gap: Distinguish cloud monitors built around structured features from the monitoring needed for prompt and output-quality drift.
Why it matters: A cloud monitor’s availability alone does not establish that it covers the LLM behavior you need to watch. - Layered safety monitoring: Describe how to combine a cloud-native safety filter with a team’s own red-team suite.
Why it matters: The serving platform’s built-in protection and your own safety tests serve complementary roles. - Managed rollback routing: Explain why rollback on major managed platforms can be a routing configuration change rather than a model load.
Why it matters: Traffic routing makes managed rollback an operational switch rather than a fresh model-loading task.
Key terms: hybrid, vendor-neutral observability, portability, registry, access controls, CI/CD, artifacts, incident management, tabular ML, prompt drift
9.7 Retraining triggers and validation gates
This section distinguishes routine model checks from the outcome-based decision to retrain.
- Outcome Based Retraining Decisions: Explain why a drift alert or scheduled review should prompt assessment rather than automatic retraining, and why the decision should depend on whether model performance has degraded.
Why it matters: Unnecessary retraining wastes compute and engineering time, while leaving a degraded model in place costs users.
Key terms: cadence, input drift, output degradation, outcome-based
9.7.1 Retrain triggers
This subsection sets the evidence threshold for retraining and explains how to respond when input drift is detected.
- Evidence Based Retraining Decisions: Explain why input drift alone does not justify retraining, and describe how to investigate a drift alert before deciding whether to retrain.
Why it matters: A drift alert calls for checking whether users are actually receiving worse outputs before committing to a retrain.
Key terms: retrain trigger, input drift, leading indicator, canary prompts, output degradation, drift thresholds
9.7.2 Incremental retraining from checkpoint
This section explains how to retrain from a production checkpoint, validate the update, and limit the risk of forgetting earlier capabilities.
- Checkpoint retraining advantage: Explain why retraining starts from the most recent checkpoint instead of the base model.
Why it matters: This makes a model refresh a smaller training job rather than a restart from the base model. - Incremental retraining and release loop: Trace how refreshed data becomes a candidate model and how its evaluation determines the next step.
Why it matters: A refreshed model should earn deployment through comparison, not merely because training completed. - Catastrophic forgetting safeguards: Explain the main risk of repeated incremental retraining and the two habits recommended to reduce it.
Why it matters: An update that improves a new domain can still damage skills the production model already had. - Self-distillation alternative and limits: Describe when SDFT is presented as an alternative and the limits on its use here.
Why it matters: SDFT is a possible response to persistent forgetting, not a default replacement for the simpler safeguards.
Key terms: most recent checkpoint, incremental retraining, delta, current production checkpoint, held-out set, register, promote, rollback, catastrophic forgetting, replay slice
9.7.3 Gating retrains
This section sets a deployment gate for retrained models and explains a research approach to preserving safety during fine-tuning.
- Retrain validation gate checks: State the checks a retrained model must pass before deployment: overall evaluation against production, safety thresholds, category-level regression limits, and canary prompt behavior.
Why it matters: A better aggregate score can hide a safety failure or a regression in one category. - Retrain decisions and failed gates: Explain when to retrain and what to do if a retrained model fails validation. Describe the section’s instruction to start from a checkpoint and run the full evaluation suite before deployment.
Why it matters: The process prevents an unvalidated retrain from replacing a known production model. - Safety gradient conflict and SPF: Explain the safety–utility gradient conflict described in the research note and how Safety-Preserving Fine-Tuning (SPF) addresses it alongside post-training safety tests.
Why it matters: It gives a reason safety can worsen during retraining and a way to constrain that risk before final testing.
Key terms: validation gate, overall eval suite, safety suite, category-level regressions, canary prompts, retrain on outcomes, checkpoint, full evaluation suite, safety-related gradients, task-utility gradients
9.8 Emerging base models and strategic planning
The section argues that reusable deployment foundations lower the cost of future models and that teams should plan across multiple horizons when deciding what to evaluate next.
- Reusable deployment foundations: Explain why a team’s later model deployments can be faster than its first and why treating each model as a one-off loses that advantage.
Why it matters: Building reusable foundations makes each subsequent deployment less costly. - Long-horizon evaluation planning: Describe what planning beyond a single model involves and why a team tracks developments in base models, techniques, and platforms.
Why it matters: A longer view helps the team choose what is worth evaluating rather than reacting to every new release.
Key terms: infrastructure, evaluation suites, data pipelines, deployment procedures, planning horizons, base models, techniques, platforms, evaluation budget
9.8.1 Planning horizons
This section lays out a phased plan for turning one monitored fine-tuned model into a repeatable, organization-wide model adaptation capability.
- First Production Lifecycle: Explain why the six-month priority is to run one fine-tuned model through its full production lifecycle before trying to scale.
Why it matters: Completing one production cycle exposes operational work that a training run alone cannot teach. - Reusable Scaling Infrastructure: Describe how the one-year plan turns a successful deployment into a repeatable process for additional use cases.
Why it matters: Shared infrastructure lets a team add use cases without rebuilding its production process each time. - Organizational Adaptation Capability: Describe what distinguishes the three-year plan from a series of individual model projects.
Why it matters: The long-term aim is to make adaptation a dependable capability that domain teams can use, not a succession of one-off efforts.
Key terms: 6-month plan, full lifecycle, rollback, retraining cycle, 1-year plan, reusable infrastructure, standardized training pipeline, shared evaluation framework, 3-year plan, organizational capability
9.8.2 Tracking the field
This section recommends selective monitoring of new base models, a quick evaluation screen before full evaluation, and delaying scale-out until after a successful deployment.
- Rapid base model screening: Explain how to decide whether a newly released base model warrants a full evaluation cycle.
Why it matters: A quick screen focuses evaluation effort on promising releases. - Selective field monitoring: Explain why both the sources monitored and the frequency of checking them matter when tracking base models.
Why it matters: A useful monitoring habit depends on where you look, not just how often you look. - Deployment before scale-out: Distinguish planning for scale from building for scale in this section's recommendation.
Why it matters: The distinction keeps scale in view without making it a prerequisite for the first successful deployment.
Key terms: canary prompts, eval suite, full evaluation cycle, base model landscape, sources, cadence, scale, successful deployment
9.9 End-to-end case study: One year of an IT support model
Introduces an illustrative IT support model deployment by showing how it is launched and monitored, and how monitoring, team decisions, and the registry work together.
- Monitoring decisions and audit trail: Explain the operational pattern illustrated by the deployment: monitoring surfaces a signal, the team decides whether to act, and the registry preserves an audit trail.
Why it matters: This separates detecting a potential problem from deciding what to do and retaining a record of the decision. - Initial launch and monitoring cadence: Describe how the team prepares and initially operates it-support-v1: register and activate the DPO-optimized model, save canary and safety baselines, then run drift detection weekly and canary prompts and the safety suite monthly.
Why it matters: The example makes the initial monitoring setup concrete without assuming a managed platform is already in place.
Key terms: monitoring, registry, audit trail, it-support-v1, DPO-optimized model, canary and safety baselines, drift detection, cron
9.9.1 When to wait, when to act
An IT support model case study shows why a drift alert warrants monitoring when canary performance holds, but action when canary results and user feedback confirm a problem.
- Corroborating drift before retraining: Explain why the team waited after the month 4 drift alert but acted in month 7. Distinguish shifted inputs from evidence that the model’s answers have deteriorated.
Why it matters: Checking whether drift affects answers helps avoid unnecessary retraining without ignoring a real decline. - Response to confirmed decline: Describe what the team did after the month 7 signals pointed to a real problem, including how it evaluated and registered the replacement model.
Why it matters: The case connects a confirmed production problem to a versioned, evaluated model update.
Key terms: drift detector, canary suite, Token-F1, two-signal discipline, updated dataset, SFT checkpoint, DPO, it-support-v2, deployment-gate evaluation
9.9.2 The rollback that proved the playbook
A production safety failure shows how monitoring, rollback, retraining, and a model registry support recovery and later audits.
- Safety alert and rollback: Trace how the safety alert led the team to investigate and roll back it-support-v2 to v1.
Why it matters: A detected safety regression is useful only if the team can quickly restore a safer version. - Cleaned retrain validation: Describe what the team changed before promoting the replacement model.
Why it matters: Rollback restores service, while cleaning the data and validating the retrain addresses the cause of the failure. - Registry history for audits: Explain how the year-end registry state helped answer a later compliance question.
Why it matters: Keeping a record of retired versions makes past production behavior easier to explain.
Key terms: monthly safety run, bias_fairness, hard alert, model_registry rollback, biased examples, it-support-v3, full safety suite, model registry, retired, active
9.10 Closing the loop
This section brings the adaptation techniques together as an operational loop that keeps a domain-specific LLM useful and reliable over time.
- Operational feedback across adaptation: Explain that the operational layer wraps every rung of the adaptation continuum: drift detection, canary prompts, and rollback provide feedback whether a team uses a few-shot prompt template or a DPO checkpoint.
Why it matters: It connects a team's chosen adaptation technique to the ongoing work of keeping it reliable. - Deployment-driven improvement: Explain why the section values honest feedback between models and users over training-pipeline sophistication, and how deployments, retrains, rollbacks, and base-model upgrades help a team improve its playbook.
Why it matters: It gives the team a reason to learn from production experience rather than treating deployment as the end of adaptation.
Key terms: adaptation continuum, operational layer, feedback loop, drift detection, canary prompts, rollback procedure, feedback loops, playbook, retrain, rollback
9.11 Summary
9.12 References
This section lists references on model drift, monitoring, safety, and continual learning without explaining those topics.
Concepts explained in this chapter
- rollbackReverting a deployed system to an earlier version when the current version causes problems.
- drift detectionA method or component that identifies changes in data or system behavior over time.
- canary promptA fixed input submitted periodically to a model so its output can be compared with a saved baseline.
- model registryA system for storing model artifacts and metadata such as versions, training details, and evaluation records for audit and deployment control.
- driftA change over time in inputs, their relationship to correct outputs, or model behavior that can degrade performance.
- concept driftA change in the relationship between a model’s input data and the concepts or outcomes it is expected to represent.
- monitoringThe ongoing observation of a deployed model to detect performance issues, failures, or drift in production.
- versioningThe practice of tracking distinct iterations of a model and their changes over time.
- drift scoreA numerical measure of the difference between reference and production text distributions, computed here from centroid cosine similarity.
- data driftA change over time in the characteristics or distribution of data a model encounters, potentially reducing its performance.
- canary checkA check of model behavior on selected prompts to detect changes or regressions in its responses.
- catastrophic forgettingThe loss of previously learned capabilities when a model is fine-tuned on narrower data, as updates overwrite parameters supporting older skills.
- cosine similarityA measure of the angle-based similarity between two vectors, often used to compare weighted text representations.
- quality gateA check that determines whether generated or collected examples meet quality requirements before they are accepted into a dataset.
- model adaptationModifying a pretrained model for a particular task, domain, or use case without training it from scratch.
- foundation modelThe original pretrained model whose weights are kept fixed while LoRA learns an additive adaptation.
- regressionA change that makes a model less likely to decline requests outside its competence.
- self-distillation fine-tuningA fine-tuning method in which a model conditioned on an expert demonstration trains the same model conditioned on the prompt alone.
- embeddingA numerical vector representation of an item that encodes features useful for computational comparison.
- test suiteAn organized collection of test cases designed to evaluate a system against defined criteria.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
