9 Managing model evolution, drift, and versioning
This chapter covers
- Versioning models and data with a lightweight JSON registry
- Detecting four types of model drift using TF-IDF cosine similarity and canary prompts
- Rolling back to a previous model version when a new deployment fails
- Monitoring security and abuse patterns with monthly red-team testing
- Upgrading base models safely without breaking fine-tuned adapters
- Maintaining and retraining on a sustainable cadence
Half a year into production, your team discovers the fine-tuned IT support model has started producing subtly different outputs. Customer satisfaction scores have dipped 8%. Nobody changed the model, so what happened? The world changed around it: users began asking about a new software platform the model had never seen, the support team updated their ticket categories, and a base-model security patch altered the underlying tokenizer behavior. Your model is the same, but the environment it serves is not.
You have trained a LoRA adapter, run full supervised fine-tuning, distilled a student model from a stronger teacher, and applied preference optimization to sharpen response quality and safety. Your model is ready for production. The hard part is over, or so it seems.