liveBook · Manning

TracedService

As explained in Designing AI Systems

TracedService is the book's base service class that automatically captures tracing context and handles telemetry-related service operations.

TracedService is the base class platform services extend to participate in distributed tracing. It places instrumentation inside service implementations, so workflow developers can call platform services without wrapping each call in tracing code. section 7.5 section 7.5.1

The class provides two context managers:

  • trace_operation creates a span for non-LLM work, such as knowledge-base search or guardrail validation. Operation-specific attributes become queryable on the span. section 7.5.1
  • trace_generation creates a generation for an LLM call, where model name, token counts, cost, and time to first token can be recorded. section 7.5.1

Both managers record timing and errors and yield a child context. Passing that context to downstream services nests their work under the current operation in the trace tree. section 7.5.1

Listing 7.6 TracedService base class with span and generation context managers
class TracedService:
    """Base class providing automatic tracing for all
    platform services."""
 
    def __init__(self, service_name: str,
                 observability: ObservabilityClient):
        self.service_name = service_name
        self.observability = observability
 
    @contextmanager
    def trace_operation(self, operation: str,
                         trace_context: TraceContext,
                         **attributes):
        span = self.observability.start_span(    #A
            trace_id=trace_context.trace_id,
            parent_span_id=trace_context.span_id,
            service=self.service_name,
            operation=f"{self.service_name}.{operation}",
            attributes=attributes)
        child_context = TraceContext(    #B
            trace_id=trace_context.trace_id,
            span_id=span.span_id,
            parent_span_id=trace_context.span_id,
            workflow_id=trace_context.workflow_id)
        try:
            yield child_context    #C
            self.observability.end_span(
                span.span_id, 
➥status=StatusCode.OK)    #D
        except Exception as e:
            self.observability.end_span(
                span.span_id,
                status=StatusCode.ERROR,
                error_message=str(e))
            raise
 
    @contextmanager
    def trace_generation(self, trace_context: TraceContext,
                          model: str, 
➥**attributes):    #E
        gen = self.observability.start_generation(
            trace_id=trace_context.trace_id,
            parent_span_id=trace_context.span_id,
            service=self.service_name,
            model=model, **attributes)
        try:
            yield gen
            self.observability.end_generation(gen.span_id)
        except Exception as e:
            self.observability.end_generation(
                gen.span_id,
                status=StatusCode.ERROR,
                error_message=str(e))
            raise

The base-class implementation shows how span and generation context managers connect service operations to the observability client. section 7.5.1

The ObservabilityClient starts and ends trace primitives through local methods, then buffers completed telemetry for batched delivery to the Observability Service. This avoids a network round-trip for each instrumented call. Separate service-specific publishers add metrics such as guardrail evaluation rates and search relevance distributions. section 7.5.1 section 7.5.2

Listing 7.7 Model Service chat method with automatic generation tracking
class ModelService(TracedService):
 
    def chat(self, request, trace_context):
        with self.trace_generation(    #A
            trace_context, model=request.model,
        ) as gen_ctx:
            provider = self._select_provider
➥(request)    #B
            response = provider.chat(
                request.messages, request.config)
            gen_ctx.update(    #C
                prompt_tokens=response.usage.prompt_tokens,
                completion_tokens=response.usage.completion_tokens,
                cost_usd=self._calculate_cost(response),
            )
            return response

A Model Service chat method uses trace_generation internally and updates generation metadata with token usage and cost. section 7.5.1

Automatic instrumentation covers standard platform service calls, not application-specific code between them. Use the SDK's trace_operation context manager to add a span for custom work, such as reranking, and avoid a gap in the trace. section 7.5.4

How Designing AI Systems covers it

How deep it goes
Explains context-manager implementation, child-context propagation, and local buffering before batched delivery.
Examples
Shows a Model Service chat method updating generation metadata and a custom reranking span added through the SDK.
What you can do after
Apply it by extending the base class and using its tracing context managers inside platform services.
How it connects
Connects distributed tracing, context propagation, generation tracking, and domain-specific metrics.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Designing AI Systems. The section links point to the parts of the book it draws on.