liveBook · Manning

Monitoring

As explained in LLM Customization and Fine-Tuning

The ongoing observation of a deployed model to detect performance issues, failures, or drift in production.

A deployed model can degrade even when its weights do not change. New user requests, revised support categories, or a base-model update can erode task accuracy and safety. Monitoring observes production behavior so teams can detect these changes and decide when to intervene. section 9.0

A practical monitoring loop combines production evidence with planned response triggers:

  • Log inputs, outputs, user feedback, and retrieved documents for retrieval-augmented generation systems. Sample the logs for weekly review; production traffic is a valuable evaluation set. section 2.10
  • Track drift with methods such as TF-IDF cosine similarity and canary prompts. Watch for domain-specific failures as well as overall accuracy changes. section 9.0
  • Run a red-team prompt suite monthly to catch new jailbreak techniques and safety regressions that a one-time deployment gate may miss. section 9.11
  • Set rollback triggers before deployment. Candidate triggers include drops in refusal rate or canary task accuracy, latency excursions, and abuse-rate spikes. section 2.10
  • Keep the previous two model versions ready for serving, with a target of less than one hour from detection to verified rollback. Retrain based on outcomes and gate the result through the full evaluation suite. section 9.11

An IT support assistant

Six months after deployment, customer satisfaction falls 8%. Users are asking about a new software platform, ticket categories have changed, and a security patch has altered tokenizer behavior. Monitoring needs to catch the assistant naming the wrong internal tool or departing from the expected format, not just a decline in overall accuracy. section 9.0

Figure 9.1 The model lifecycle: train, deploy, monitor, then either retrain or roll back.

The lifecycle places monitoring after deployment and connects it to retraining or rollback. section 9.0

How LLM Customization and Fine-Tuning covers it

How deep it goes
Specifies logging, weekly review, rollback triggers, and monthly red-team testing, and names drift-detection methods.
Examples
Uses an IT support assistant whose satisfaction falls as requests, ticket categories, and tokenizer behavior change.
What you can do after
Operate monitoring through production log review, safety testing, predefined rollback triggers, and outcome-based retraining.
How it connects
Connects monitoring to model drift, retrieval-augmented generation, regression testing, rollback, and retraining.

Also explained in

Compare how 51 books explain monitoring →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.