A model can pass its offline evaluation and still fail users after deployment. Incoming data changes, software breaks, and prediction quality can deteriorate. Monitoring makes these problems visible early enough to investigate and respond. Start with software health, then watch data quality, model quality, and business outcomes; good model metrics cannot compensate for an unreliable service. section 14.0 section 14.6
Logs provide the context behind a metric or alert. Record events and predictions with timestamps. When feasible and affordable, retain each prediction’s input features and model output; storing every output may be too costly. Include identifiers that let an operator connect records from the same user, session, or request. A log entry should say what happened, not merely that something happened. section 14.2 section 14.5.2
❌logger.info("Fraud probability calculated")✅logger.info(f"Fraud probability={score:.3f}foruserid{user_id}")
Compare a message that only announces a calculation with one that records the score and user ID needed to investigate it. section 14.2
Build a monitoring plan around questions someone will need to answer during an incident:
- Is the service working? Track a small set of service-level indicators, such as request volume, errors, latency, uptime, and resource use. Connect alerts to the existing incident process and prepare guidance for expected failures. section 14.2 section 13.3.1
- Are inputs and outputs changing? Check missing data, schema and feature statistics, and prediction drift. If ground truth arrives later, use data and prediction drift as interim signals rather than treating them as proof of model quality. section 14.5.1 section 14.4.3
- Is the model still useful? Compare quality metrics with historical baselines and examine important segments. Choose monitoring granularity according to risk; an alert may lead to retraining, rebuilding, or a backup strategy. section 14.4.3 section 14.6
Two systems, different records
A retail forecasting plan stores prediction features, outputs, and timestamps in ClickHouse, while Kafka, Prometheus, and Grafana support operational statistics and visualization. A stock-photo search plan instead logs the original query, candidate IDs, final top-20 ranking, failures, and fallbacks; user, session, and request IDs make a search traceable. Follow existing company storage where it fits rather than inventing a separate logging platform. section 14.5.1 section 14.5.2 section 13.3.1
Monitoring also matters before release. Watch running experiments for failures, but do not quietly exclude users or segments unless that option was part of the initial design. Reliable reporting depends on keeping results distinguishable: overwritten experiment results can turn an apparent improvement into a misleading report. Decide in advance what to retain, what should trigger an alert, and who will investigate. section 12.3 section 13.3.1
