liveBook · Manning

Model quality

As explained in Machine Learning System Design

The degree to which a machine learning model produces accurate, reliable, and appropriate outputs for its intended task.

Model quality describes whether a machine learning model continues to produce accurate, relevant, and valuable outputs for its intended task. Unlike deterministic software, a model can degrade when live data differs from training data, so quality must be monitored after release. section 14.0 section 14.6

Treat model quality as an operational monitoring loop:

  • Log each prediction with its input features, output, and timestamp so quality changes can be investigated. section 14.5.1
  • Track performance metrics and compare them with historical baselines to detect model decay early. section 14.6
  • Check data quality and drift because changed feature distributions or invalid inputs can undermine predictions. section 14.5.1 section 14.6
  • Respond to deterioration by retraining on new data or building a model for a newly changed segment. section 14.6

Demand forecasting at Supermegaretail

A demand-forecasting system records predictions in a column-oriented database and combines Evidently AI with Kafka, Prometheus, and Grafana. The monitoring plan tracks model outputs, latency, errors, prediction statistics, data quality, and drift, creating evidence for deciding whether the model remains reliable. section 14.5.1 section 14.6

How Machine Learning System Design covers it

How deep it goes
Connects quality to logging, performance baselines, drift, retraining, and replacement without detailing metric formulas.
Examples
Uses demand forecasting, prediction logs, Evidently AI, and historical performance baselines as monitoring examples.
What it assumes
Assumes a deployed ML system receiving changing real-world data and contrasts it with deterministic software.
What you can do after
Diagnose model decay by comparing performance with historical baselines and respond through retraining or model replacement.
How it connects
Links model quality with data quality, data drift, concept drift, software health, model relevance, and business KPIs.
Cover of Machine Learning System Design
liveBookmanning.com
Read Machine Learning System Design on liveBook Get the book at manning.com

This explainer was generated by Manning's knowledge graph from the text of Machine Learning System Design. The section links point to the parts of the book it draws on.