A deployed model can degrade even when its weights do not change. New user requests, revised support categories, or a base-model update can erode task accuracy and safety. Monitoring observes production behavior so teams can detect these changes and decide when to intervene. section 9.0
A practical monitoring loop combines production evidence with planned response triggers:
- Log inputs, outputs, user feedback, and retrieved documents for retrieval-augmented generation systems. Sample the logs for weekly review; production traffic is a valuable evaluation set. section 2.10
- Track drift with methods such as TF-IDF cosine similarity and canary prompts. Watch for domain-specific failures as well as overall accuracy changes. section 9.0
- Run a red-team prompt suite monthly to catch new jailbreak techniques and safety regressions that a one-time deployment gate may miss. section 9.11
- Set rollback triggers before deployment. Candidate triggers include drops in refusal rate or canary task accuracy, latency excursions, and abuse-rate spikes. section 2.10
- Keep the previous two model versions ready for serving, with a target of less than one hour from detection to verified rollback. Retrain based on outcomes and gate the result through the full evaluation suite. section 9.11
An IT support assistant
Six months after deployment, customer satisfaction falls 8%. Users are asking about a new software platform, ticket categories have changed, and a security patch has altered tokenizer behavior. Monitoring needs to catch the assistant naming the wrong internal tool or departing from the expected format, not just a decline in overall accuracy. section 9.0
Figure 9.1 The model lifecycle: train, deploy, monitor, then either retrain or roll back.

The lifecycle places monitoring after deployment and connects it to retraining or rollback. section 9.0


















































