liveBook · Manning

Chapter 7: The Observability Service: seeing what your AI does

What you will learn in chapter 7 of Designing AI Systems

This introduction frames the Observability Service as the platform component that unifies telemetry across services so teams can understand AI behavior, cost, performance, and quality.

After this chapter you will be able to

What it builds on

Section by section

Introduction

This introduction frames the Observability Service as the platform component that unifies telemetry across services so teams can understand AI behavior, cost, performance, and quality.

Key terms: Observability Service, unified visibility, queryable, telemetry, logs, metrics, traces, quality scores, cost attribution, Experimentation Service

7.1 Why AI systems need specialized observability

This section establishes why AI systems require observability that measures whether outputs are actually useful and correct, not merely whether infrastructure is operating.

Key terms: traditional monitoring, AI observability, infrastructure metrics, output quality, response times, error rates, latency charts, error rate dashboards, healthy, 200 OK

7.1.1 From infrastructure health to output quality

This section explains why AI observability must extend infrastructure monitoring with continuous measurement of response quality and AI-specific pipeline signals.

Key terms: traditional monitoring, deterministic, non-deterministic outputs, output quality, latency, error rates, AI observability, first-class signal, continuous measurement, traditional monitoring tools

7.1.2 Cross-service correlation

This section explains why end-to-end debugging of AI requests requires correlating telemetry across services with a shared trace ID.

Key terms: cross-service correlation, end-to-end behavior, local telemetry, shared trace ID, service hop, distributed tracing, isolated log streams, timestamps, synchronized clocks

7.1.3 Cost, and the bridge to experimentation

This section explains why AI cost observability must aggregate variable token-based request costs, relate spending to quality, and provide the shared data foundation for experimentation.

Key terms: token consumption, context window utilization, model choice, caching effectiveness, per-request costs, Model Service, Observability Service, aggregated budget views, spending by team and workflow, projections

7.2 The observability data model

Introduces a question-driven approach to defining the observability data model before specifying service contracts.

Key terms: observability data model, AI platform teams, abstractions, data model, service contracts, Observability Service

7.2.1 Deriving the model from questions

Derives an AI observability data model from operational questions using sessions, traces, spans, generations, and scores.

Key terms: session, trace, span, generation, score, distributed tracing, parent span, child span, waterfall, status

7.2.2 How the primitives connect: tracing a single request

This section traces one patient request to show how sessions, traces, spans, generations, and scores are created, correlated, and queried by the Observability Service.

Key terms: session, trace, trace_id, span, API Gateway, Session Service, generation, first-class attributes, GPT-4o, prompt tokens

7.2.3 Logs and metrics: complementing the request-level model

This section explains how structured logs and aggregate metrics complement trace-centric request data by exposing detailed events and system-wide operational patterns.

Key terms: structured logs, log entries, span, span_id, trace_id, span attributes, aggregate metrics, counters, histograms, time-series data

7.3 The Observability Service contract

This section defines the Observability Service’s gRPC contract as the interface for ingesting, querying, and managing AI telemetry without duplicating session ownership.

Key terms: Observability Service, gRPC contract, telemetry, ingestion, querying, cost and budget views, logging, metrics, tracing, generations

7.3.1 Walking through the contract

This section explains how the Observability Service contract separates telemetry ingestion and querying into specialized operation groups for logs, metrics, traces, scores, costs, budgets, and platform-wide health.

Key terms: IngestLogs, QueryLogs, RecordMetrics, QueryMetrics, structured logs, time-series data, RecordSpan, RecordGeneration, GetTrace, QueryTraces

7.3.2 Ingestion vs query patterns

This subsection distinguishes high-throughput, nonblocking telemetry ingestion from interactive observability queries and explains why ingestion must never affect user-facing requests.

Key terms: ingestion operations, query operations, data receiver, fire-and-forget, hot path, user-facing request processing, single point of failure, request path, Observability Service outage, observability client

7.4 Structured logging, metrics, and distributed tracing

This section introduces structured logs, aggregate metrics, and distributed traces as the three core telemetry types the Observability Service uses for debugging AI systems.

Key terms: structured logs, aggregate metrics, distributed traces, telemetry, Observability Service, structured logging, metrics, distributed tracing, debugging, data structures

7.4.1 Structured logging for AI-specific debugging

This section explains how a common structured-log format captures AI-specific context, correlates events across services, supports investigation queries, and requires privacy-aware retention management.

Key terms: structured logging, common log event structure, attributes, numeric_attributes, Model Service, Data Service, trace_id, span_id, trace view, span

7.4.2 Metrics collection

This section explains how the platform collects, names, queries, visualizes, and alerts on aggregate AI-system metrics.

Key terms: counters, histograms, dimensional labels, MODEL_COST_USD, PlatformMetrics, ai.platform.{service}.{metric_name}, _total, _ms, _score, heavy-tailed distributions

7.4.3 Distributed tracing and the debugging workflow

This section explains how distributed traces connect AI-service work into a request view and how to debug production problems by moving from traces to logs to metrics.

Key terms: distributed trace, span, root span, child span, trace_id, parent_span_id, Generation, Trace, RecordGeneration, RecordSpan

7.5 How platform services report telemetry

This section introduces how the platform makes tracing, AI-specific telemetry, and structured logging automatic for workflow developers through shared instrumentation layers.

Key terms: distributed tracing, domain-specific metrics, structured logging, SDK, TracedService, trace_operation, trace_generation, span, generation, context propagation

7.5.1 TracedService: automatic instrumentation

This section explains how platform services use TracedService to automatically create, correlate, buffer, and report tracing telemetry without requiring workflow developers to instrument ordinary service calls.

Key terms: TracedService, trace_operation, trace_generation, span, generation, time to first token, child_context, span_id, parent_span_id, trace tree

7.5.2 Domain-specific telemetry

This section explains how each platform service publishes its existing domain data as labeled aggregate metrics, complementing per-request tracing with operational visibility over time.

Key terms: domain-specific telemetry, publishing layer, ObservabilityClient, PlatformMetrics, observability by default, record_counter, record_histogram, MODEL_REQUEST_TOTAL, MODEL_COST_USD, MODEL_REQUEST_DURATION

7.5.3 The observability client: batching and buffering

This section explains how the ObservabilityClient delivers telemetry efficiently and resiliently by batching buffered records and flushing them both on demand and on a timer.

Key terms: ObservabilityClient, Observability Service, TracedService, spans, metrics, logs, end_span, in-memory span buffer, batch_size, _flush_spans

7.5.4 Adding custom observability beyond the defaults

This section explains how workflow developers instrument application-specific logic that automatic platform observability does not cover.

Key terms: trace operation context manager, custom span, TracedService, platform-generated spans, trace ID, span tree, span_id, ObservabilityClient, structured logs, custom metrics

7.6 Quality scores and cost attribution

This section introduces quality scores as telemetry attached to AI request traces, spans, or generations to judge whether responses are actually good and to support later experimentation.

Key terms: scores, quality signals, traces, spans, generations, traditional monitoring

7.6.1 Scores: measuring response quality

This section explains how to measure AI response quality with layered scoring sources, link scores to telemetry, and use quality signals to find traces for improvement.

Key terms: automated scorers, LLM-as-judge scorers, human reviewers, annotation queues, calibration, ground truth labels, Score, trace_id, span_id, generation_id

7.6.2 Cost attribution and budget tracking

This section explains how the Observability Service attributes AI costs across dimensions, supports progressive cost drill-downs, and warns teams about projected budget overruns.

Key terms: per-request costs, dimensional reports, get_cost_report, drill-down, team, workflow_id, budget alerts, progressive thresholds, scope, period

7.7 Putting it all together

This section assembles the Observability Service into an automatically instrumented, queryable platform with specialized stores, operational dashboards, and traces that feed an asynchronous improvement loop.

Key terms: TracedService, metrics publishers, observability client, spans, generations, telemetry, Log Store, Time-Series DB, Trace + Generation Store, Score Store

7.8 Summary

This summary recaps how the Observability Service captures AI-specific telemetry without slowing requests, then connects traces, logs, metrics, quality, and cost into operational debugging and control.

Where this is used later

Concepts explained in this chapter

← Chapter 6: Tools and Guardrails: Enabling safe, managed AI actions Chapter 8: The Experimentation Service: improving what your AI does →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.