liveBook · Manning

AI-specific metric

As explained in Designing AI Systems

A quantitative measurement describing an AI application's usage, quality, or conversational behavior.

AI-specific metrics extend infrastructure monitoring with measurements that reveal whether an AI system is useful, safe, appropriately routed, and economically sustainable. Uptime, latency, and errors can remain normal while response quality or retrieval relevance deteriorates. section 7.1.1

Production observability captures dimensions such as:

  • Token usage and cost attribution, exposing consumption and spending patterns. section 7.1.1
  • Model-selection visibility, revealing unexpected routing to cheaper or less capable models. section 7.1.1
  • Retrieval-quality metrics, indicating whether returned document chunks provide relevant context. section 7.1.1
  • Safety-filter activations and response-quality scoring, distinguishing real problems from false positives and weak answers. section 7.1.1
Figure 7.3 AI observability extends traditional monitoring with dimensions unique to AI systems. Traditional metrics (uptime, latency, errors) still matter but miss the questions that AI operations teams need answered. Token tracking, model selection visibility, retrieval quality, safety evaluations, cost attribution, and response quality scoring are all AI-specific concerns that the platform must capture.

Maps traditional monitoring metrics to AI-specific dimensions, including token tracking, model selection, retrieval quality, safety evaluation, cost attribution, and response quality. section 7.1.1

How Designing AI Systems covers it

How deep it goes
Identifies important metric dimensions and failure scenarios but stops short of defining formulas, schemas, or implementation procedures.
Examples
Uses unexpected model routing, irrelevant document chunks, safety-filter activations, token tracking, and response-quality scoring as examples.
What you can do after
Diagnose hidden quality degradation despite healthy uptime, latency, and error measurements.
How it connects
Connects observability with monitoring, retrieval quality, safety evaluation, cost attribution, model selection, and response quality.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Designing AI Systems. The section links point to the parts of the book it draws on.