Chapter 7: The Observability Service: seeing what your AI does
This introduction frames the Observability Service as the platform component that unifies telemetry across services so teams can understand AI behavior, cost, performance, and quality.
After this chapter you will be able to
- Unified platform visibility. Explain how the Observability Service turns isolated signals from platform services into a unified, queryable view of an individual AI request and its behavior.
- Observability and experimentation boundary. Distinguish passive observability from active experimentation: observability watches and makes production data available, while experimentation changes something and measures its impact.
- AI observability gap. Explain the fundamental gap between traditional application monitoring and AI observability: conventional monitoring can show that a system is technically running while an AI assistant is functioning poorly for users.
- AI health definition. Explain why a successful HTTP response and fast response time do not establish AI health: an assistant can return 200 OK with a fluent and confident but completely wrong answer.
- Limits of traditional monitoring. Explain why latency, error rates, and uptime remain necessary but cannot by themselves establish whether an AI application produced a useful response. Contrast deterministic applications, where healthy infrastructure usually implies correct output, with AI applications whose outputs can vary for the same inputs.
- Output quality as telemetry. Describe why AI observability treats output quality as a first-class signal that must be measured continuously for each response, rather than assuming correctness from the code or infrastructure state. State that doing this requires data that traditional monitoring tools do not capture.
- Cross-service request correlation. Explain why a single AI request must be correlated across multiple platform services to understand its end-to-end behavior and latency.
- Shared trace ID. Describe how a shared trace ID follows a request across every service hop and enables its telemetry to be connected into one trace.
- AI cost variability. Explain why AI application cost varies more from request to request than traditional web-server operating cost. Describe the AI-specific factors named here: token consumption, conversation length, context-window utilization, model choice, and caching effectiveness.
- Aggregated budget visibility. Describe the two layers of AI cost visibility: Model Service per-request cost detail for investigating anomalies, and Observability Service aggregation for budgets, spending trends, and projections by team and workflow.
- Cost-quality experimentation link. Explain why cost data alone cannot show whether an AI workflow is worthwhile, and how quality scores attached to production traffic enable cost-versus-quality comparisons. Describe the division of responsibility: Observability and Experimentation are separate services that share traces and scores as a data foundation for systematic optimization.
- Question-driven data-model design. Explain why the observability data model is derived from the questions AI platform teams need to answer rather than from a predefined list of abstractions.
- Observability primitive hierarchy. Describe how sessions, traces, spans, generations, and scores model AI application activity, including their containment and attachment relationships.
- Trace and span request reconstruction. Explain how a trace and its nested spans reconstruct one request's path across services, including timing, status, inputs and outputs, cost, and metadata.
- Generation-specific telemetry. Justify generations as a specialized span type for LLM calls, and describe the model-specific fields that make direct cost, latency, and model-behavior queries practical.
- Request trace hierarchy. Describe how a single request is organized under a session and trace, with service operations recorded as correlated spans. Explain that the API Gateway creates a unique trace_id and every participating service uses it so the platform can reconstruct the end-to-end request.
- Generation telemetry distinction. Distinguish a model generation from a regular span. Describe how a generation records first-class, directly indexable model-specific attributes such as model, token counts, cost, time to first token, and cache-hit status.
- Asynchronous quality score attachment. Explain how quality scores are added to a completed trace asynchronously to assess whether an answer was good. Describe that scores identify their source, such as a heuristic, a model judge, or a human reviewer.
- Structured log correlation. Explain how structured logs capture discrete, detailed events during a span and correlate those events with the enclosing span and request trace using span_id and trace_id.
- Aggregate metric monitoring. Describe aggregate metrics as independently recorded counters and histograms that combine numeric measurements across many requests into time-series data for dashboards and alerts.
- Observability contract responsibilities. Describe the three broad responsibilities of the Observability Service contract: ingesting telemetry from platform services, querying it for dashboards and debugging, and managing derived cost and budget views.
- Six telemetry contract concerns. State the six concerns covered by the fourteen contract operations: logging, metrics, tracing, scores, cost management, and health monitoring; distinguish what each captures at a high level.
- Log and metric separation. Distinguish the log and metric operation pairs and explain why the contract keeps them separate. Describe logs as individually stored, field-searchable event records and metrics as aggregate numeric time series queried over aggregation windows.
- Trace assembly operations. Describe the tracing operations and trace assembly model: services ingest spans and LLM generations under a shared trace ID, while the Observability Service later assembles and returns complete traces. Distinguish retrieving one trace with GetTrace from searching many with QueryTraces.
- Independent quality score lifecycle. Explain why RecordScore and QueryScores are separate from trace operations. Describe scores as attachments that can arrive over time from automated heuristics, LLM evaluators, or human reviewers and can be queried without retrieving full traces.
- Cost and budget operations. Describe how the contract turns per-request generation costs into operational cost management. Identify GetCostReport for drill-down by workflow, model, and time period, SetBudgetAlert for spending thresholds and notifications, and GetBudgetStatus for current spending against those thresholds.
- Ingestion and query separation. Distinguish the Observability Service's ingestion operations from its query operations, including their different traffic directions, purposes, and performance constraints.
- Telemetry failure isolation. Explain why observability publishing must not block or fail a user-facing request, and describe the required behavior when the Observability Service is temporarily unavailable.
- Core telemetry categories. State the three core telemetry types collected by the Observability Service: structured logs, aggregate metrics, and distributed traces, and identify their respective focus.
- Complementary debugging views. Explain how structured logs, aggregate metrics, and distributed traces work together during debugging by combining event detail, system-wide trends, and a cross-service request view.
- AI-specific structured log context. Describe what makes the platform's structured logs useful for AI debugging: they use a common log event structure with standard correlation IDs plus queryable AI-specific attributes. Give examples such as model names, token counts, cost estimates, fallback chains, guardrail results, or retrieval relevance scores.
- Trace-correlated cross-service debugging. Explain how trace_id and span_id connect log events across services for one request. Trace the debugging workflow of starting from a trace view, finding a problematic span, and retrieving the events emitted during that span instead of manually matching disconnected logs by timestamp.
- Metric types and dimensions. Describe the platform's two metric types—counters for cumulative totals and histograms for distributions—and explain how dimensional labels such as provider, model, and workflow ID enable focused queries.
- Distribution-based AI monitoring. Justify using histograms and percentile views instead of averages for latency and retrieval-quality metrics, especially because AI response times and relevance can have misleading distributions.
- Distributed trace structure. Describe a distributed trace as a tree of spans that follows one request across platform services. Explain that a root span represents the user-visible request, child spans represent nested service operations, and trace context links them through trace_id and parent_span_id.
- Three-level debugging workflow. Trace the debugging workflow for a slow AI request: start with traces to locate the bottleneck, inspect logs for that trace_id to find the causal event sequence, and use metrics to determine whether the problem is isolated or systemic. Illustrate how a slow Model Service span can lead to trace-linked logs revealing provider timeouts and fallback overhead, while metrics confirm a broader provider error-rate spike.
- Default observability experience. Explain how a workflow developer receives distributed tracing, domain-specific metrics, and structured logging by calling platform services through the SDK without writing observability code.
- Telemetry reporting layers. Describe the three layers that report platform telemetry: TracedService for automatic trace primitives, per-service domain telemetry, and ObservabilityClient for asynchronous delivery.
- TracedService context managers. Describe how TracedService provides trace_operation for non-LLM work and trace_generation for LLM calls. Explain that both record timing and errors, while generations additionally capture LLM-specific information such as model name, token counts, cost, and time to first token.
- Nested trace context propagation. Trace how trace_operation yields a child context that downstream service calls pass forward, allowing each new span or generation to attach to its immediate parent. Distinguish this from trace_generation, which yields a generation object rather than a child context because generations are leaf nodes.
- Buffered telemetry reporting. Explain that TracedService calls local ObservabilityClient methods such as start_span and end_span, which create and finalize telemetry in an in-memory buffer. Describe how completed spans and generations are later flushed in batches to the Observability Service through RecordSpan or RecordGeneration RPCs, keeping tracing off the request critical path.
- Observability by default. Explain how instrumentation belongs inside platform service backends so ordinary workflow calls automatically produce telemetry. Describe examples: Model Service chat creates a generation, Guardrails records checked policies and actions, and Data Service search records the index, result count, and top relevance score; developers add custom spans only when needed.
- Domain telemetry publishing layer. Describe the thin publishing layer that each service uses to translate domain data it already produces into the platform metric format through the ObservabilityClient. Explain that this layer is a small adapter rather than a new source of business data.
- Counter and histogram selection. Distinguish record_counter from record_histogram when publishing service metrics. State that counters accumulate values such as request totals and cost, while histograms collect individual observations such as latency so distributions and percentiles can be calculated.
- Labeled aggregate metric queries. Explain how metric labels such as provider, model, workflow_id, and cache_hit make aggregate metrics queryable by dimensions. Describe how these dimensions let operations teams isolate questions such as p95 latency for a model in one workflow or total cost for cache misses over a time window.
- Observability client responsibilities. Describe how each platform service uses an ObservabilityClient to connect to the Observability Service and deliver spans, metrics, and logs without sending each telemetry record directly over the network.
- Batch flushing workflow. Trace how a completed span is finalized, appended to an in-memory buffer, and sent in a RecordSpan gRPC batch when the buffer reaches batch_size. Describe how periodic flushing also sends spans, metrics, and logs when buffers have not filled.
- Failure buffering and data limits. Explain how a failed flush puts its batch back at the front of the buffer for a later retry, while max_buffer_size drops the oldest data to prevent unbounded memory growth. Distinguish this best-effort telemetry behavior from client-facing request handling: a telemetry outage should not fail or slow the request.
- Custom span instrumentation. Explain why application-specific workflow steps need explicit instrumentation and describe how the SDK trace operation context manager creates a custom span around such logic.
- Quality scores for AI responses. Explain why operational telemetry alone cannot determine whether an AI response was good, and describe scores as quality signals attached to traces, spans, or generations. State that scores make it possible to tell whether changes improve the system.
- Layered quality scoring. Explain the three sources of quality scores—automated scorers, LLM-as-judge scorers, and human reviewers—and how their cost, scalability, accuracy, and role differ. Describe how human labels calibrate the automated and LLM-based layers.
- Score telemetry data model. Describe how a Score ties a quality signal to a trace and, when needed, to a span or generation. Explain that its name, source, and one-of numeric, boolean, or categorical value identify what was measured, who produced it, and the appropriate kind of result.
- Dimensional cost drill-downs. Explain how the Observability Service turns per-request costs into dimensional cost reports and uses progressive drill-downs to identify the team, workflow, and model driving spend.
- Budget alert configuration. Describe how budget alerts use scoped spending limits, time periods, notification channels, and progressive thresholds to warn teams before they exceed a budget.
- Automatic telemetry instrumentation. Explain how TracedService, metrics publishers, and the observability client produce complete telemetry without developers adding tracing wrappers to workflow code. Describe the telemetry captured and how local batching keeps it off the user-facing latency path.
- Specialized telemetry storage. Describe how the Log Store, Time-Series DB, Trace + Generation Store, and Score Store divide observability data according to their query patterns. Explain that the stores use different storage engines because debugging events, time-range aggregates, hierarchical traces, and quality signals need different access patterns.
- Operational dashboard feedback loop. Describe the operations, quality, and cost dashboards and the kinds of signals each presents. Explain how dashboard charts and alerts use aggregate telemetry but link operators back to underlying traces for request-level debugging.
What it builds on
- Observability: Cost tracking and metrics (chapter 3) · The model service already tracks cost and metrics as telemetry for observability.
- Vector storage and search (chapter 5) · Search orchestration provides retrieved context whose relevance can affect generated answers.
- Guardrails in practice: input, output and behavioral policies (chapter 6) · Input, output, and behavioral guardrails can shape what an AI system returns.
- Enabling informed decisions (chapter 3) · Model-service telemetry supports informed decisions about AI behavior and operations.
- Routing strategies (chapter 3) · Routing strategies include model-selection decisions that may change which model handles a request.
- What the model service tracks (chapter 3) · The Model Service tracks model-related usage and cost information.
- Chunking: breaking text into retrievable pieces (chapter 5) · Chunking creates the retrievable document pieces whose relevance affects context quality.
- Search orchestration (chapter 5) · Search orchestration returns context that can be assessed for retrieval quality.
Section by section
Introduction
This introduction frames the Observability Service as the platform component that unifies telemetry across services so teams can understand AI behavior, cost, performance, and quality.
- Unified platform visibility: Explain how the Observability Service turns isolated signals from platform services into a unified, queryable view of an individual AI request and its behavior.
Why it matters: Unified visibility lets teams diagnose an AI application's behavior without manually piecing together separate service logs. - Operational observability questions: State the four operational questions the Observability Service is designed to answer: what happened, how long it took, how much it cost, and how good it was.
Why it matters: These questions define the practical outcomes that the chapter's telemetry infrastructure must support. - Observability and experimentation boundary: Distinguish passive observability from active experimentation: observability watches and makes production data available, while experimentation changes something and measures its impact.
Why it matters: Keeping measurement separate from intervention makes it clear how production visibility becomes the foundation for systematic improvement.
Key terms: Observability Service, unified visibility, queryable, telemetry, logs, metrics, traces, quality scores, cost attribution, Experimentation Service
7.1 Why AI systems need specialized observability
This section establishes why AI systems require observability that measures whether outputs are actually useful and correct, not merely whether infrastructure is operating.
- AI observability gap: Explain the fundamental gap between traditional application monitoring and AI observability: conventional monitoring can show that a system is technically running while an AI assistant is functioning poorly for users.
Why it matters: A healthy AI system must be judged by more than uptime, latency, and error rates. - Output quality signals: Describe the kinds of user-visible AI failures that may not appear in infrastructure metrics, including less helpful responses, needing extra turns to resolve simple questions, and asking for information the user already gave.
Why it matters: These symptoms reveal quality failures that users experience even when dashboards show successful requests. - AI health definition: Explain why a successful HTTP response and fast response time do not establish AI health: an assistant can return 200 OK with a fluent and confident but completely wrong answer.
Why it matters: Defining health in terms of outcome quality prevents teams from mistaking a successful API call for a successful AI interaction.
Key terms: traditional monitoring, AI observability, infrastructure metrics, output quality, response times, error rates, latency charts, error rate dashboards, healthy, 200 OK
7.1.1 From infrastructure health to output quality
This section explains why AI observability must extend infrastructure monitoring with continuous measurement of response quality and AI-specific pipeline signals.
- Limits of traditional monitoring: Explain why latency, error rates, and uptime remain necessary but cannot by themselves establish whether an AI application produced a useful response. Contrast deterministic applications, where healthy infrastructure usually implies correct output, with AI applications whose outputs can vary for the same inputs.
Why it matters: A healthy AI service can still give users poor answers, so operational health is not the same as application quality. - Output quality as telemetry: Describe why AI observability treats output quality as a first-class signal that must be measured continuously for each response, rather than assuming correctness from the code or infrastructure state. State that doing this requires data that traditional monitoring tools do not capture.
Why it matters: Measuring quality continuously makes it possible to detect degradation that ordinary availability and performance dashboards cannot reveal. - AI-specific diagnostic signals: Describe the AI-specific signals that reveal hidden degradation: token consumption, model-selection and fallback decisions, retrieval quality, and safety-filter activations. Explain what each can expose, such as prompt bloat and cost growth, lower-capability fallback routing, irrelevant retrieved context, or guardrail false positives.
Why it matters: These signals identify problems that can remain invisible when latency, uptime, and error rates appear normal.
Key terms: traditional monitoring, deterministic, non-deterministic outputs, output quality, latency, error rates, AI observability, first-class signal, continuous measurement, traditional monitoring tools
7.1.2 Cross-service correlation
This section explains why end-to-end debugging of AI requests requires correlating telemetry across services with a shared trace ID.
- Cross-service request correlation: Explain why a single AI request must be correlated across multiple platform services to understand its end-to-end behavior and latency.
Why it matters: A slow or failing AI interaction can originate in any service it traverses, not only the model call. - Shared trace ID: Describe how a shared trace ID follows a request across every service hop and enables its telemetry to be connected into one trace.
Why it matters: The shared identifier replaces unreliable manual matching of separate service records with a coherent request-level view. - Manual log correlation limits: Describe the manual debugging problem that occurs without a shared trace ID: comparing timestamps across isolated log streams and relying on synchronized clocks.
Why it matters: This limitation explains why distributed tracing is a required observability capability rather than a convenience.
Key terms: cross-service correlation, end-to-end behavior, local telemetry, shared trace ID, service hop, distributed tracing, isolated log streams, timestamps, synchronized clocks
7.1.3 Cost, and the bridge to experimentation
This section explains why AI cost observability must aggregate variable token-based request costs, relate spending to quality, and provide the shared data foundation for experimentation.
- AI cost variability: Explain why AI application cost varies more from request to request than traditional web-server operating cost. Describe the AI-specific factors named here: token consumption, conversation length, context-window utilization, model choice, and caching effectiveness.
Why it matters: Understanding what drives variable AI cost makes unusual requests and rising spend explainable. - Aggregated budget visibility: Describe the two layers of AI cost visibility: Model Service per-request cost detail for investigating anomalies, and Observability Service aggregation for budgets, spending trends, and projections by team and workflow.
Why it matters: Teams need both the ability to investigate an expensive request and the ability to manage spending before a budget limit is reached. - Cost-quality experimentation link: Explain why cost data alone cannot show whether an AI workflow is worthwhile, and how quality scores attached to production traffic enable cost-versus-quality comparisons. Describe the division of responsibility: Observability and Experimentation are separate services that share traces and scores as a data foundation for systematic optimization.
Why it matters: A lower-cost model is only an improvement if quality remains acceptable, so optimization needs evidence about both outcomes and spending. - Platform outcome questions: State the three platform questions this section uses to assess observability completeness: how much it costs, whether it is working well, and whether it is getting better.
Why it matters: These questions provide a concise checklist for judging whether an AI observability system covers spending, quality, and improvement.
Key terms: token consumption, context window utilization, model choice, caching effectiveness, per-request costs, Model Service, Observability Service, aggregated budget views, spending by team and workflow, projections
7.2 The observability data model
Introduces a question-driven approach to defining the observability data model before specifying service contracts.
- Question-driven data-model design: Explain why the observability data model is derived from the questions AI platform teams need to answer rather than from a predefined list of abstractions.
Why it matters: Starting from operational questions helps ensure the observability model captures information teams can actually use. - Data model before contracts: Describe the sequencing decision to establish the underlying observability data model before diving into Observability Service contracts.
Why it matters: A shared model gives later service interfaces a consistent foundation for the telemetry they ingest and expose.
Key terms: observability data model, AI platform teams, abstractions, data model, service contracts, Observability Service
7.2.1 Deriving the model from questions
Derives an AI observability data model from operational questions using sessions, traces, spans, generations, and scores.
- Observability primitive hierarchy: Describe how sessions, traces, spans, generations, and scores model AI application activity, including their containment and attachment relationships.
Why it matters: This hierarchy lets you move from a problematic conversation to the exact request and operation that caused it. - Trace and span request reconstruction: Explain how a trace and its nested spans reconstruct one request's path across services, including timing, status, inputs and outputs, cost, and metadata.
Why it matters: When an answer is slow or wrong, the trace tree shows which service step failed or became the bottleneck. - Generation-specific telemetry: Justify generations as a specialized span type for LLM calls, and describe the model-specific fields that make direct cost, latency, and model-behavior queries practical.
Why it matters: Treating LLM calls as first-class records makes the expensive and slowest part of many AI requests searchable and optimizable. - Session-level conversation analysis: Explain why sessions group multiple request traces for a conversation and identify the conversation-level measures that a single trace cannot provide.
Why it matters: Session grouping reveals whether an entire conversation was costly, prolonged, or degraded as context accumulated. - Quality score signals: Describe scores as explicit quality signals attached to traces, spans, or generations, including their sources, value forms, and role in connecting observability to improvement work.
Why it matters: A successful, fast request is not necessarily helpful or correct, so scores make response quality measurable.
Key terms: session, trace, span, generation, score, distributed tracing, parent span, child span, waterfall, status
7.2.2 How the primitives connect: tracing a single request
This section traces one patient request to show how sessions, traces, spans, generations, and scores are created, correlated, and queried by the Observability Service.
- Request trace hierarchy: Describe how a single request is organized under a session and trace, with service operations recorded as correlated spans. Explain that the API Gateway creates a unique trace_id and every participating service uses it so the platform can reconstruct the end-to-end request.
Why it matters: A shared trace_id lets you follow one AI request across multiple platform services instead of debugging each service in isolation. - Generation telemetry distinction: Distinguish a model generation from a regular span. Describe how a generation records first-class, directly indexable model-specific attributes such as model, token counts, cost, time to first token, and cache-hit status.
Why it matters: Making model behavior and cost first-class telemetry enables operational questions that generic service timing alone cannot answer. - Asynchronous quality score attachment: Explain how quality scores are added to a completed trace asynchronously to assess whether an answer was good. Describe that scores identify their source, such as a heuristic, a model judge, or a human reviewer.
Why it matters: Trace timing and cost show what happened, but attached scores provide evidence about whether the response was actually useful or correct.
Key terms: session, trace, trace_id, span, API Gateway, Session Service, generation, first-class attributes, GPT-4o, prompt tokens
7.2.3 Logs and metrics: complementing the request-level model
This section explains how structured logs and aggregate metrics complement trace-centric request data by exposing detailed events and system-wide operational patterns.
- Structured log correlation: Explain how structured logs capture discrete, detailed events during a span and correlate those events with the enclosing span and request trace using span_id and trace_id.
Why it matters: Structured logs let teams investigate why a service produced a particular outcome, not merely whether it succeeded or how long it ran. - Aggregate metric monitoring: Describe aggregate metrics as independently recorded counters and histograms that combine numeric measurements across many requests into time-series data for dashboards and alerts.
Why it matters: Metrics reveal fleet-wide trends and incidents, such as rising p99 latency or an error spike, that no single request trace can establish. - Complementary observability layers: Distinguish the roles of traces and spans, structured logs, and aggregate metrics: request-level work breakdown and outcome, detailed event-level reasons, and cross-request patterns over time.
Why it matters: Using the right telemetry layer prevents teams from trying to answer fleet-wide questions with one trace or detailed debugging questions with a dashboard.
Key terms: structured logs, log entries, span, span_id, trace_id, span attributes, aggregate metrics, counters, histograms, time-series data
7.3 The Observability Service contract
This section defines the Observability Service’s gRPC contract as the interface for ingesting, querying, and managing AI telemetry without duplicating session ownership.
- Observability contract responsibilities: Describe the three broad responsibilities of the Observability Service contract: ingesting telemetry from platform services, querying it for dashboards and debugging, and managing derived cost and budget views.
Why it matters: The contract makes observability usable both for collecting AI-system evidence and for investigating or governing it later. - Six telemetry contract concerns: State the six concerns covered by the fourteen contract operations: logging, metrics, tracing, scores, cost management, and health monitoring; distinguish what each captures at a high level.
Why it matters: These concerns organize the service interface around evidence of what happened, how the system performed, how well it worked, what it cost, and whether services are healthy. - Session ownership and correlation: Explain why sessions are absent from the Observability Service contract: the Session Service owns their lifecycle, while traces carry a session_id that groups a conversation’s traces and enables session-level metrics such as total cost or turn count.
Why it matters: Referencing session IDs preserves cross-conversation analysis without duplicating session management across services.
Key terms: Observability Service, gRPC contract, telemetry, ingestion, querying, cost and budget views, logging, metrics, tracing, generations
7.3.1 Walking through the contract
This section explains how the Observability Service contract separates telemetry ingestion and querying into specialized operation groups for logs, metrics, traces, scores, costs, budgets, and platform-wide health.
- Log and metric separation: Distinguish the log and metric operation pairs and explain why the contract keeps them separate. Describe logs as individually stored, field-searchable event records and metrics as aggregate numeric time series queried over aggregation windows.
Why it matters: Using distinct data models lets the platform support both detailed debugging and efficient operational measurement. - Trace assembly operations: Describe the tracing operations and trace assembly model: services ingest spans and LLM generations under a shared trace ID, while the Observability Service later assembles and returns complete traces. Distinguish retrieving one trace with GetTrace from searching many with QueryTraces.
Why it matters: This design makes it possible to follow one request across independently reporting platform services. - Generation-specific telemetry: Explain why RecordGeneration is separate from RecordSpan. State that generations contain LLM-specific fields such as model name, token counts, cost, and time to first token, enabling direct indexing and filtering of generation records.
Why it matters: Separating generations makes LLM cost and performance questions efficient to answer without scanning all spans. - Independent quality score lifecycle: Explain why RecordScore and QueryScores are separate from trace operations. Describe scores as attachments that can arrive over time from automated heuristics, LLM evaluators, or human reviewers and can be queried without retrieving full traces.
Why it matters: Quality assessment often happens after a request, so the platform must support scores that accumulate independently of immutable request traces. - Cost and budget operations: Describe how the contract turns per-request generation costs into operational cost management. Identify GetCostReport for drill-down by workflow, model, and time period, SetBudgetAlert for spending thresholds and notifications, and GetBudgetStatus for current spending against those thresholds.
Why it matters: Cost telemetry becomes actionable when teams can allocate spending, detect spikes, and compare usage with budgets. - Platform-wide service health: Distinguish GetServiceHealth from a service's own liveness endpoint. Explain that it derives a cross-platform health view from observed telemetry patterns, such as recent spans, error-rate spikes, and unusual ingestion volume.
Why it matters: Operations dashboards need evidence about how services are behaving across the platform, not merely whether individual processes are alive.
Key terms: IngestLogs, QueryLogs, RecordMetrics, QueryMetrics, structured logs, time-series data, RecordSpan, RecordGeneration, GetTrace, QueryTraces
7.3.2 Ingestion vs query patterns
This subsection distinguishes high-throughput, nonblocking telemetry ingestion from interactive observability queries and explains why ingestion must never affect user-facing requests.
- Ingestion and query separation: Distinguish the Observability Service's ingestion operations from its query operations, including their different traffic directions, purposes, and performance constraints.
Why it matters: This separation keeps operational telemetry collection from competing with either application requests or human investigation workflows. - Telemetry failure isolation: Explain why observability publishing must not block or fail a user-facing request, and describe the required behavior when the Observability Service is temporarily unavailable.
Why it matters: Observability is operationally critical, but making application availability depend on it would turn it into a single point of failure. - Buffered telemetry retry: Describe how a calling service's observability client handles temporary ingestion failures through internal batching, periodic flushing, local buffering, and retry on a later cycle.
Why it matters: This approach preserves application availability while recovering telemetry after a brief observability outage, with only a possible small data gap.
Key terms: ingestion operations, query operations, data receiver, fire-and-forget, hot path, user-facing request processing, single point of failure, request path, Observability Service outage, observability client
7.4 Structured logging, metrics, and distributed tracing
This section introduces structured logs, aggregate metrics, and distributed traces as the three core telemetry types the Observability Service uses for debugging AI systems.
- Core telemetry categories: State the three core telemetry types collected by the Observability Service: structured logs, aggregate metrics, and distributed traces, and identify their respective focus.
Why it matters: These three telemetry types provide complementary views of how an AI system behaves. - Complementary debugging views: Explain how structured logs, aggregate metrics, and distributed traces work together during debugging by combining event detail, system-wide trends, and a cross-service request view.
Why it matters: Debugging AI systems requires both broad evidence of a trend and detailed evidence about an individual request. - Telemetry implementation preview: Describe this section's scope as an examination of telemetry data structures, naming conventions, and joint debugging use, and distinguish it from section 7.5, which explains automatic production and delivery by platform services.
Why it matters: Separating telemetry design from telemetry production clarifies what the observability layer records and how services later report it.
Key terms: structured logs, aggregate metrics, distributed traces, telemetry, Observability Service, structured logging, metrics, distributed tracing, debugging, data structures
7.4.1 Structured logging for AI-specific debugging
This section explains how a common structured-log format captures AI-specific context, correlates events across services, supports investigation queries, and requires privacy-aware retention management.
- AI-specific structured log context: Describe what makes the platform's structured logs useful for AI debugging: they use a common log event structure with standard correlation IDs plus queryable AI-specific attributes. Give examples such as model names, token counts, cost estimates, fallback chains, guardrail results, or retrieval relevance scores.
Why it matters: AI-specific fields turn generic service logs into evidence for diagnosing AI behavior and operations. - Trace-correlated cross-service debugging: Explain how trace_id and span_id connect log events across services for one request. Trace the debugging workflow of starting from a trace view, finding a problematic span, and retrieving the events emitted during that span instead of manually matching disconnected logs by timestamp.
Why it matters: Correlation IDs let an operator investigate a slow or failing AI request as one cross-service flow. - Log query patterns: Describe the main log-query pattern of filtering by trace_id to retrieve all events for one request, and explain how attribute filters enable cross-request investigations such as finding every evaluation of a named guardrail policy. State why cursor-based pagination is used for log queries that can match millions of events.
Why it matters: Effective query patterns make large volumes of platform logs usable during focused investigations. - Privacy-aware log retention: Explain why user_id may be pseudonymized in regulated environments and describe the retention approach: detailed logs for 30 days, aggregate summaries for 90 days, and critical audit events for the period required by the compliance framework. Justify retention policies as necessary to prevent continuously growing log storage costs.
Why it matters: Logs spread to downstream stores and exports, so privacy and storage lifecycle decisions must be designed from the start.
Key terms: structured logging, common log event structure, attributes, numeric_attributes, Model Service, Data Service, trace_id, span_id, trace view, span
7.4.2 Metrics collection
This section explains how the platform collects, names, queries, visualizes, and alerts on aggregate AI-system metrics.
- Metric types and dimensions: Describe the platform's two metric types—counters for cumulative totals and histograms for distributions—and explain how dimensional labels such as provider, model, and workflow ID enable focused queries.
Why it matters: Metric types and labels let operators answer operational questions without manually inspecting individual events. - Metric naming conventions: Explain the naming scheme ai.platform.{service}.{metric_name}, including how service namespaces avoid collisions and how the _total, _ms, and _score suffixes indicate metric type.
Why it matters: Consistent names make the metric catalog discoverable and allow people and tools to identify metrics without memorizing every definition. - Distribution-based AI monitoring: Justify using histograms and percentile views instead of averages for latency and retrieval-quality metrics, especially because AI response times and relevance can have misleading distributions.
Why it matters: Averages can conceal slow tail requests or a meaningful share of poor retrieval results that users actually experience. - Metric dashboards and alerts: Describe the standard metric views and threshold-based alerts the platform supports, including the distinction between monitoring trends and triggering investigation or response when conditions cross thresholds.
Why it matters: Dashboards provide rapid operational visibility, while alerts turn concerning metric patterns into action for operators and safety reviewers.
Key terms: counters, histograms, dimensional labels, MODEL_COST_USD, PlatformMetrics, ai.platform.{service}.{metric_name}, _total, _ms, _score, heavy-tailed distributions
7.4.3 Distributed tracing and the debugging workflow
This section explains how distributed traces connect AI-service work into a request view and how to debug production problems by moving from traces to logs to metrics.
- Distributed trace structure: Describe a distributed trace as a tree of spans that follows one request across platform services. Explain that a root span represents the user-visible request, child spans represent nested service operations, and trace context links them through trace_id and parent_span_id.
Why it matters: A trace lets an operator see where time was spent in one slow AI request rather than inspecting isolated service events. - Generation and trace assembly: Explain why a Generation wraps a Span and adds model-specific fields such as model, provider, token counts, cost, cache status, and time to first token. Describe that services record individual spans and generations, while the Observability Service correlates them by trace_id to assemble a Trace, so there is no RecordTrace operation.
Why it matters: This design supports efficient model-specific investigation while preserving one complete view of each request. - Three-level debugging workflow: Trace the debugging workflow for a slow AI request: start with traces to locate the bottleneck, inspect logs for that trace_id to find the causal event sequence, and use metrics to determine whether the problem is isolated or systemic. Illustrate how a slow Model Service span can lead to trace-linked logs revealing provider timeouts and fallback overhead, while metrics confirm a broader provider error-rate spike.
Why it matters: Using correlated traces, logs, and metrics turns a multi-service production investigation from manual timestamp matching into a focused diagnosis.
Key terms: distributed trace, span, root span, child span, trace_id, parent_span_id, Generation, Trace, RecordGeneration, RecordSpan
7.5 How platform services report telemetry
This section introduces how the platform makes tracing, AI-specific telemetry, and structured logging automatic for workflow developers through shared instrumentation layers.
- Default observability experience: Explain how a workflow developer receives distributed tracing, domain-specific metrics, and structured logging by calling platform services through the SDK without writing observability code.
Why it matters: Automatic observability lets developers diagnose AI workflows without manually adding instrumentation to every service call. - Telemetry reporting layers: Describe the three layers that report platform telemetry: TracedService for automatic trace primitives, per-service domain telemetry, and ObservabilityClient for asynchronous delivery.
Why it matters: These layers separate universal tracing, service-specific signals, and efficient telemetry delivery so observability is comprehensive without slowing requests. - Service-specific telemetry: State examples of domain-specific telemetry added by platform services: Model Service token counts and cost, Guardrails Service policy evaluation results, and Data Service relevance scores.
Why it matters: Service-specific signals reveal AI behavior that generic request timing and error data cannot explain.
Key terms: distributed tracing, domain-specific metrics, structured logging, SDK, TracedService, trace_operation, trace_generation, span, generation, context propagation
7.5.1 TracedService: automatic instrumentation
This section explains how platform services use TracedService to automatically create, correlate, buffer, and report tracing telemetry without requiring workflow developers to instrument ordinary service calls.
- TracedService context managers: Describe how TracedService provides trace_operation for non-LLM work and trace_generation for LLM calls. Explain that both record timing and errors, while generations additionally capture LLM-specific information such as model name, token counts, cost, and time to first token.
Why it matters: Choosing the right tracing primitive makes an AI request understandable at both the service-operation and individual-model-call levels. - Nested trace context propagation: Trace how trace_operation yields a child context that downstream service calls pass forward, allowing each new span or generation to attach to its immediate parent. Distinguish this from trace_generation, which yields a generation object rather than a child context because generations are leaf nodes.
Why it matters: Passing context through service calls lets the Observability Service reconstruct one coherent tree for a request that crosses many services. - Buffered telemetry reporting: Explain that TracedService calls local ObservabilityClient methods such as start_span and end_span, which create and finalize telemetry in an in-memory buffer. Describe how completed spans and generations are later flushed in batches to the Observability Service through RecordSpan or RecordGeneration RPCs, keeping tracing off the request critical path.
Why it matters: Local buffering allows detailed observability without making every user request wait for a remote telemetry RPC. - Observability by default: Explain how instrumentation belongs inside platform service backends so ordinary workflow calls automatically produce telemetry. Describe examples: Model Service chat creates a generation, Guardrails records checked policies and actions, and Data Service search records the index, result count, and top relevance score; developers add custom spans only when needed.
Why it matters: Centralized automatic instrumentation gives every workflow consistent tracing while removing repetitive tracing work from application developers.
Key terms: TracedService, trace_operation, trace_generation, span, generation, time to first token, child_context, span_id, parent_span_id, trace tree
7.5.2 Domain-specific telemetry
This section explains how each platform service publishes its existing domain data as labeled aggregate metrics, complementing per-request tracing with operational visibility over time.
- Domain telemetry publishing layer: Describe the thin publishing layer that each service uses to translate domain data it already produces into the platform metric format through the ObservabilityClient. Explain that this layer is a small adapter rather than a new source of business data.
Why it matters: Thin publishers let every service contribute useful operational telemetry without duplicating its domain logic. - Counter and histogram selection: Distinguish record_counter from record_histogram when publishing service metrics. State that counters accumulate values such as request totals and cost, while histograms collect individual observations such as latency so distributions and percentiles can be calculated.
Why it matters: Choosing the right metric type determines whether operators can measure totals and throughput or diagnose tail-latency behavior. - Labeled aggregate metric queries: Explain how metric labels such as provider, model, workflow_id, and cache_hit make aggregate metrics queryable by dimensions. Describe how these dimensions let operations teams isolate questions such as p95 latency for a model in one workflow or total cost for cache misses over a time window.
Why it matters: Labels turn platform-wide measurements into actionable answers about a particular model, workflow, or request condition. - Aggregate metrics and tracing: Distinguish the role of aggregate domain metrics from automatic tracing by TracedService. Explain that tracing captures what happened on an individual request, whereas publishers show how the system behaves across many requests over time for dashboards and alerts.
Why it matters: Both request-level debugging and fleet-level operational monitoring are needed to run AI systems reliably.
Key terms: domain-specific telemetry, publishing layer, ObservabilityClient, PlatformMetrics, observability by default, record_counter, record_histogram, MODEL_REQUEST_TOTAL, MODEL_COST_USD, MODEL_REQUEST_DURATION
7.5.3 The observability client: batching and buffering
This section explains how the ObservabilityClient delivers telemetry efficiently and resiliently by batching buffered records and flushing them both on demand and on a timer.
- Observability client responsibilities: Describe how each platform service uses an ObservabilityClient to connect to the Observability Service and deliver spans, metrics, and logs without sending each telemetry record directly over the network.
Why it matters: This client keeps telemetry delivery from adding network latency to normal AI request handling. - Batch flushing workflow: Trace how a completed span is finalized, appended to an in-memory buffer, and sent in a RecordSpan gRPC batch when the buffer reaches batch_size. Describe how periodic flushing also sends spans, metrics, and logs when buffers have not filled.
Why it matters: Batching reduces network overhead, while timed delivery keeps dashboards current even when traffic is sparse. - Failure buffering and data limits: Explain how a failed flush puts its batch back at the front of the buffer for a later retry, while max_buffer_size drops the oldest data to prevent unbounded memory growth. Distinguish this best-effort telemetry behavior from client-facing request handling: a telemetry outage should not fail or slow the request.
Why it matters: The design preserves application availability during observability outages while placing a deliberate bound on memory use and possible telemetry loss.
Key terms: ObservabilityClient, Observability Service, TracedService, spans, metrics, logs, end_span, in-memory span buffer, batch_size, _flush_spans
7.5.4 Adding custom observability beyond the defaults
This section explains how workflow developers instrument application-specific logic that automatic platform observability does not cover.
- Custom span instrumentation: Explain why application-specific workflow steps need explicit instrumentation and describe how the SDK trace operation context manager creates a custom span around such logic.
Why it matters: Custom spans prevent developer-written steps from becoming unexplained gaps in an end-to-end request trace. - Trace continuity for custom logic: Describe how a custom span joins the existing trace by inheriting its trace ID and nesting in the span tree, and state how custom metrics and structured logs can be correlated with that span.
Why it matters: Correlated spans, metrics, and logs let a developer investigate custom workflow behavior in the same context as platform activity.
Key terms: trace operation context manager, custom span, TracedService, platform-generated spans, trace ID, span tree, span_id, ObservabilityClient, structured logs, custom metrics
7.6 Quality scores and cost attribution
This section introduces quality scores as telemetry attached to AI request traces, spans, or generations to judge whether responses are actually good and to support later experimentation.
- Quality scores for AI responses: Explain why operational telemetry alone cannot determine whether an AI response was good, and describe scores as quality signals attached to traces, spans, or generations. State that scores make it possible to tell whether changes improve the system.
Why it matters: A fast, error-free AI request can still be unhelpful, wrong, or off topic, so quality must be observed separately from operational health.
Key terms: scores, quality signals, traces, spans, generations, traditional monitoring
7.6.1 Scores: measuring response quality
This section explains how to measure AI response quality with layered scoring sources, link scores to telemetry, and use quality signals to find traces for improvement.
- Layered quality scoring: Explain the three sources of quality scores—automated scorers, LLM-as-judge scorers, and human reviewers—and how their cost, scalability, accuracy, and role differ. Describe how human labels calibrate the automated and LLM-based layers.
Why it matters: Layering score sources gives teams broad quality coverage without paying for human or model-based review on every request. - Score telemetry data model: Describe how a Score ties a quality signal to a trace and, when needed, to a span or generation. Explain that its name, source, and one-of numeric, boolean, or categorical value identify what was measured, who produced it, and the appropriate kind of result.
Why it matters: A well-linked score lets teams analyze quality at the request, service-call, or individual-generation level rather than treating quality as an unconnected metric. - Scorer implementation and discovery: Describe how the KeyElementScorer computes a completeness ratio by checking expected response elements, while the LLMJudgeScorer sends the original question and response to a judge model for a criterion-specific score. Explain why LLM judges should use a different model from the evaluated model and how scores enable quality-filtered trace queries.
Why it matters: These scorer patterns turn quality into searchable operational evidence that can expose poor responses for investigation and improvement.
Key terms: automated scorers, LLM-as-judge scorers, human reviewers, annotation queues, calibration, ground truth labels, Score, trace_id, span_id, generation_id
7.6.2 Cost attribution and budget tracking
This section explains how the Observability Service attributes AI costs across dimensions, supports progressive cost drill-downs, and warns teams about projected budget overruns.
- Dimensional cost drill-downs: Explain how the Observability Service turns per-request costs into dimensional cost reports and uses progressive drill-downs to identify the team, workflow, and model driving spend.
Why it matters: Cost drill-downs let teams move from an unexpected total bill to the specific workload and model that can be optimized. - Budget alert configuration: Describe how budget alerts use scoped spending limits, time periods, notification channels, and progressive thresholds to warn teams before they exceed a budget.
Why it matters: Early, scoped alerts give teams time to reduce or optimize spending before an overage becomes a surprise invoice. - Projected spending and hard limits: Distinguish projected-spending budget alerts from hard spending enforcement, and state why hard limits should use the Model Service's rate limiting rather than automatically blocking requests from an observability alert.
Why it matters: This distinction prevents a cost-monitoring feature from unexpectedly breaking production applications while still enabling firm controls when required.
Key terms: per-request costs, dimensional reports, get_cost_report, drill-down, team, workflow_id, budget alerts, progressive thresholds, scope, period
7.7 Putting it all together
This section assembles the Observability Service into an automatically instrumented, queryable platform with specialized stores, operational dashboards, and traces that feed an asynchronous improvement loop.
- Automatic telemetry instrumentation: Explain how TracedService, metrics publishers, and the observability client produce complete telemetry without developers adding tracing wrappers to workflow code. Describe the telemetry captured and how local batching keeps it off the user-facing latency path.
Why it matters: Automatic instrumentation gives every workflow consistent traces, metrics, quality context, and cost data without making application code harder to write. - Specialized telemetry storage: Describe how the Log Store, Time-Series DB, Trace + Generation Store, and Score Store divide observability data according to their query patterns. Explain that the stores use different storage engines because debugging events, time-range aggregates, hierarchical traces, and quality signals need different access patterns.
Why it matters: Separating telemetry by query pattern makes both detailed debugging and aggregate operational analysis practical at platform scale. - Operational dashboard feedback loop: Describe the operations, quality, and cost dashboards and the kinds of signals each presents. Explain how dashboard charts and alerts use aggregate telemetry but link operators back to underlying traces for request-level debugging.
Why it matters: Dashboards and alerts turn collected telemetry into timely action while preserving a path from a trend or alert to the requests that caused it. - Synchronous request trace: Trace the synchronous patient-intake request path from context retrieval through data search, guardrail validation, model generation, and response return. Explain that each service records telemetry along this latency-sensitive path and the Observability Service assembles it into one complete trace.
Why it matters: A complete cross-service trace lets teams understand a user request as one flow rather than isolated service calls.
Key terms: TracedService, metrics publishers, observability client, spans, generations, telemetry, Log Store, Time-Series DB, Trace + Generation Store, Score Store
7.8 Summary
This summary recaps how the Observability Service captures AI-specific telemetry without slowing requests, then connects traces, logs, metrics, quality, and cost into operational debugging and control.
Where this is used later
- The Experimentation Service contract (chapter 8) · Introduces the separate Experimentation Service built on the visibility supplied by observability.
- Evaluation: datasets, offline scoring, online scoring, and human review (chapter 8) · Uses observable data for offline and online evaluation and human review.
- Making Claw better: experimentation (chapter 10) · Applies experimentation to improve the Claw assistant.
- (chapter intro) (chapter 8) · Why AI systems need observability beyond infrastructure health.
- Target lifecycle and evaluation (chapter 8) · Targets and evaluation judge whether an AI system produces desirable outcomes.
- Seeing what Claw does: observability (chapter 10) · The completed assistant needs observability that reveals what it did and whether its responses were useful.
- Online evaluation: scoring production traffic in real-time (chapter 8) · Online evaluation compares production outcomes where model choice and operating cost may vary.
- A/B testing infrastructure (chapter 8) · A/B testing infrastructure supports comparisons such as whether a more expensive model produces better outcomes.
Concepts explained in this chapter
- spanA span is a distributed-tracing record representing one timed operation within a trace and its relationship to parent and child operations.
- generationA generation is an observability record that wraps a span and adds model, provider, token, cost, cache, and first-token timing information for a model call.
- trajectoryA trace is a correlated record of a request's journey through a system, including its operations, timing, metadata, and nested work units across components.
- qualityA score is a numerical or categorical value representing how well an observed operation or output performed against an evaluation criterion.
- ObservabilityClientObservabilityClient is an SDK client that buffers spans, logs, and metrics locally and flushes them to the Observability Service in batches.
- TracedServiceTracedService is the book's base service class that automatically captures tracing context and handles telemetry-related service operations.
- observability serviceA service that collects and exposes operational information about application behavior, including latency, usage, and performance.
- correlation idA correlation ID is an identifier carried across related records or service calls so they can be associated with the same request or operation.
- telemetryOperational data emitted by a software system to represent its behavior and health, such as logs, metrics, traces, and generation records.
- budget alertA budget alert is a notification triggered when tracked or projected spending crosses a defined budget threshold.
- histogramA histogram metric records observations in value ranges so their distribution and percentile statistics can be calculated.
- observabilityThe ability to infer a system's internal behavior and health from its emitted telemetry and operational records.
- automated scoringAn automated scorer is a non-generative evaluation procedure that computes response-quality scores from recorded metrics or rules.
- sessionA session is a structured record representing one conversation between a user and an AI application, including its messages and identifying metadata.
- LLM-as-a-judgeAn LLM judge is a language model used to evaluate another model's output against specified criteria.
- trace contextDistributed trace context is propagation metadata that associates work performed across services with the same distributed trace.
- dimensional labelA metric label is a named attribute attached to a metric that enables grouping and querying by dimensions such as provider, model, or workflow.
- time seriesA time series is an ordered sequence of timestamped values showing how measurements change over time.
- logThe recording of application or system events, diagnostics, and operational information.
- trace_generationtrace_generation is a TracedService context-manager operation that creates and manages a generation trace primitive for a language-model call.
- trace_operationtrace_operation is a context-manager method that records a non-LLM service operation as a tracing span and yields context for nested calls.
- counterA counter metric is a monotonically accumulating measurement of events or quantities over time.
- trace treeA span tree is the hierarchical structure that represents nesting and parent-child relationships among spans in a trace.
- continuous improvementAn asynchronous improvement loop evaluates completed interactions after serving them and feeds the resulting scores into systematic system improvement.
- time-series databaseA time-series database stores measurements indexed by time and supports time-range queries over metrics.
- human-in-the-loopHuman review is the inspection and validation of an item by a person before it is accepted for an intended purpose.
- aggregate metricAn aggregate metric is a system-level measurement computed by summarizing observations across multiple events, requests, or components.
- batchingBatching is the accumulation of multiple items so they can be processed or transmitted together.
- dimensionDimensionality is the number of scalar values composing a vector representation.
- AI-specific metricA quantitative measurement describing an AI application's usage, quality, or conversational behavior.
- monitoringThe observation of service telemetry to assess the operational condition of a platform or system.
- experimentation-as-a-service platformAn experimentation service is a platform service that provides infrastructure for running controlled experiments and measuring the effects of application changes.
- token countToken usage is the quantity of language-model input and output tokens consumed by an AI operation.
- gRPC contractA gRPC contract is a formal service interface that defines the operations and interaction structure exposed between gRPC services.
- relevanceA retrieval relevance distribution is the statistical distribution of relevance scores produced for retrieved results across queries.
- liveness checkA liveness probe is a health-check interface that indicates whether a running service instance is alive and able to be managed by infrastructure.
- labelA ground truth label is a reference judgment assigned to an example output, typically by human annotation, for evaluating other scoring methods.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
