liveBook · Manning

Quantile metric

As explained in Machine Learning System Design

A performance measure that evaluates forecasts for a specified quantile of a predictive distribution.

A quantile metric evaluates forecast quality at selected points in a predictive distribution. In retail, it can reflect the business costs of stockouts and inventory, and support forecasts aimed at meeting service-level targets. section 5.3.1

Practical choices include:

  • Evaluate several quantiles, with or without SKU-price weights and extra penalties for under- or overforecasting. Compare absolute and percentage errors before choosing a final form. section 5.3.1
  • Monitor weighted and unweighted quantile metrics as true sales labels arrive, alongside RMSE and MAE. Set final thresholds after observing initial data and model performance. section 14.5.1

How Machine Learning System Design covers it

How deep it goes
Compares weighted, percentage, and penalized variants, while leaving the final metric form and monitoring thresholds to experiments.
Examples
Uses quantiles 1.5, 25, 50, 75, 95, and 99, with SKU-price weights and sales labels for monitoring.
What you can do after
Evaluate forecast performance across selected quantiles, optionally weighting items by SKU price and monitoring results over time.
How it connects
Links quantile metrics to quantile forecasts, stock control, loss functions, and model-quality monitoring.
Cover of Machine Learning System Design
liveBookmanning.com
Read Machine Learning System Design on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Machine Learning System Design. The section links point to the parts of the book it draws on.