liveBook · Manning

Monitoring

As explained in Designing AI Systems

The observation of service telemetry to assess the operational condition of a platform or system.

Monitoring observes service telemetry to determine whether a platform is operating acceptably. Production systems need more than inference code: they require visibility into system health, model accuracy, response behavior, and operational conditions. section 1.2.1 section 1.0

Traditional monitoring answers whether servers are available and responses are fast. AI systems also require signals about correctness: an assistant can return successful responses quickly while becoming less helpful, repeating questions, or producing confident errors. section 7.1

A production monitoring design separates several observable concerns:

  • Logging records what happened and supports querying events. section 7.3
  • Metrics measure quantities and frequencies, while health monitoring evaluates platform-wide service condition from telemetry patterns. section 7.3
  • Tracing follows request paths; generations preserve individual language-model calls as first-class records. section 7.3
  • Scores capture how well the system worked, while cost management exposes the financial view. section 7.3

A misleadingly healthy deployment

After a new patient-intake prompt is deployed, latency and error rates remain normal, yet patients need more turns and repeat information. Monitoring must correlate infrastructure signals with quality signals rather than treating successful HTTP responses as proof of health. section 7.1

Listing 7.1 Observability Service gRPC definition
syntax = "proto3";
package ai.platform.observability.v1;
 
service ObservabilityService {
  rpc IngestLogs(IngestLogsRequest) 
➥returns (IngestLogsResponse);    #A
  rpc QueryLogs(QueryLogsRequest) returns 
➥(QueryLogsResponse);    #A
 
  rpc RecordMetrics(RecordMetricsRequest) 
➥returns (RecordMetricsResponse);    #B
  rpc QueryMetrics(QueryMetricsRequest) returns 
➥(QueryMetricsResponse);    #B
 
  rpc RecordSpan(RecordSpanRequest) returns 
➥(RecordSpanResponse);    #C
  rpc RecordGeneration(RecordGenerationRequest)
      returns (RecordGenerationResponse);    #D
  rpc GetTrace(GetTraceRequest) returns (Trace);    #E
  rpc QueryTraces(QueryTracesRequest) 
➥returns (QueryTracesResponse);    #E
 
  rpc RecordScore(RecordScoreRequest) 
➥returns (RecordScoreResponse);    #F
  rpc QueryScores(QueryScoresRequest) 
➥returns (QueryScoresResponse);    #F
 
  rpc GetCostReport(CostReportRequest) 
➥returns (CostReport);    #G
  rpc SetBudgetAlert(SetBudgetAlertRequest)
      returns (SetBudgetAlertResponse);    #G
  rpc GetBudgetStatus(GetBudgetStatusRequest) 
➥returns (BudgetStatus);    #G
 
  rpc GetServiceHealth(ServiceHealthRequest)
      returns (ServiceHealthResponse);    #H
}

The Observability Service exposes monitoring through a gRPC contract for ingesting and querying logs, metrics, spans, generations, and scores. Sessions remain managed elsewhere and are linked through session_id. section 7.3

How Designing AI Systems covers it

How deep it goes
Distinguishes traditional health metrics from AI correctness and reaches a service-level contract for telemetry ingestion and querying.
Examples
Uses support-ticket classification, a degraded patient-intake prompt, and unpredictable prototype latency to expose monitoring gaps.
What it assumes
Assumes distributed production systems and basic application monitoring while teaching why AI correctness requires additional signals.
What you can do after
Diagnose production behavior through dashboards, debugging queries, correlated telemetry, and platform-wide health views.
How it connects
Connects monitoring to observability, logging, metrics, tracing, generations, scores, cost management, and sessions.

Also explained in

Compare how 51 books explain monitoring →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Designing AI Systems. The section links point to the parts of the book it draws on.