liveBook · Manning

ObservabilityClient

As explained in Designing AI Systems

ObservabilityClient is an SDK client that buffers spans, logs, and metrics locally and flushes them to the Observability Service in batches.

ObservabilityClient buffers spans, logs, and metrics locally and delivers them to the Observability Service in batches. Each platform service holds a client, separating telemetry publication from network delivery so individual events do not require a request-path round-trip. section 7.5.3 section 7.8

TracedService uses local client methods to start and end spans and generations. The context managers record timing and errors and propagate child contexts; completed spans and generations are later delivered through RecordSpan and RecordGeneration. Service-specific attributes make spans queryable. section 7.5.1

Listing 7.6 TracedService base class with span and generation context managers
class TracedService:
    """Base class providing automatic tracing for all
    platform services."""
 
    def __init__(self, service_name: str,
                 observability: ObservabilityClient):
        self.service_name = service_name
        self.observability = observability
 
    @contextmanager
    def trace_operation(self, operation: str,
                         trace_context: TraceContext,
                         **attributes):
        span = self.observability.start_span(    #A
            trace_id=trace_context.trace_id,
            parent_span_id=trace_context.span_id,
            service=self.service_name,
            operation=f"{self.service_name}.{operation}",
            attributes=attributes)
        child_context = TraceContext(    #B
            trace_id=trace_context.trace_id,
            span_id=span.span_id,
            parent_span_id=trace_context.span_id,
            workflow_id=trace_context.workflow_id)
        try:
            yield child_context    #C
            self.observability.end_span(
                span.span_id, 
➥status=StatusCode.OK)    #D
        except Exception as e:
            self.observability.end_span(
                span.span_id,
                status=StatusCode.ERROR,
                error_message=str(e))
            raise
 
    @contextmanager
    def trace_generation(self, trace_context: TraceContext,
                          model: str, 
➥**attributes):    #E
        gen = self.observability.start_generation(
            trace_id=trace_context.trace_id,
            parent_span_id=trace_context.span_id,
            service=self.service_name,
            model=model, **attributes)
        try:
            yield gen
            self.observability.end_generation(gen.span_id)
        except Exception as e:
            self.observability.end_generation(
                gen.span_id,
                status=StatusCode.ERROR,
                error_message=str(e))
            raise

See how the tracing context manager uses the client and constructs a child trace context. section 7.5.1

The delivery mechanism balances network overhead, timeliness, and bounded memory:

  • Finalizing a span adds it to an in-memory buffer; reaching the configured batch size triggers a flush of up to that many spans. section 7.5.3
  • A failed span batch returns to the front of the buffer for a later retry. If the buffer exceeds its maximum size, the oldest data is dropped. section 7.5.3
  • A background timer flushes spans, metrics, and logs even when buffers are not full, supporting low-traffic delivery and retries. section 7.5.3
Listing 7.9 ObservabilityClient batching and buffering
class ObservabilityClient:
    def __init__(self, channel,
                 batch_size: int = 100,
                 flush_interval_seconds: float = 5.0,
                 max_buffer_size: int = 10_000):
        self._stub = ObservabilityServiceStub(channel)
        self._span_buffer: list[Span] = []    #A
        self._metric_buffer: list[MetricRecord] = []    #A
        self._log_buffer: list[LogEvent] = []    #A
        self._batch_size = batch_size
        self._max_buffer_size = max_buffer_size
        self._flush_interval = flush_interval_seconds
        self._flush_thread = threading.Thread(    #B
            target=self._periodic_flush, daemon=True)    #B
        self._flush_thread.start()
 
    def end_span(self, span_id: str,
                 status: StatusCode = StatusCode.OK,
                 error_message: str = None):
        span = self._finalize_span(
            span_id, status, error_message)
        self._span_buffer.append(span)
        if len(self._span_buffer) >= self._batch_size:
            self._flush_spans()    #C
 
    def _flush_spans(self):
        if not self._span_buffer:
            return
        batch = self._span_buffer[:self._batch_size]
        self._span_buffer = (
            self._span_buffer[self._batch_size:])
        try:
            self._stub.RecordSpan(    #D
                RecordSpanRequest(spans=batch))    #D
        except Exception:
            if (len(self._span_buffer)
                    < self._max_buffer_size):
                self._span_buffer = (    #E
                    batch + self._span_buffer)    #E
            # else: drop oldest data to prevent
            # unbounded memory growth
 
    def _periodic_flush(self):
        while True:
            time.sleep(self._flush_interval)
            self._flush_spans()    #F
            self._flush_metrics()    #F
            self._flush_logs()    #F

Inspect the client’s buffers, delivery configuration, and background flush mechanism. section 7.5.3

Publishing model metrics

A Model Service publisher translates request metrics into a request counter, a cost counter, and a duration histogram. Provider, model, workflow ID, and cache-hit labels make the resulting metrics queryable by dimension. section 7.5.2

Automatic instrumentation covers platform service operations. For application logic outside those services, developers can add a custom span through the SDK, publish custom metrics through the client, and emit structured logs referencing the current span ID. section 7.5.4

How Designing AI Systems covers it

How deep it goes
Explains local instrumentation and batch delivery, including threshold and timed flushes, retries, and bounded-buffer data loss.
Examples
Uses a model metrics publisher and a custom retrieval reranking span to illustrate automatic and application-specific telemetry.
What you can do after
Evaluate and understand how buffered telemetry delivery handles network overhead, retries, low traffic, and data loss.
How it connects
Connects distributed tracing and context propagation with structured logging, counters, histograms, and telemetry ingestion.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Designing AI Systems. The section links point to the parts of the book it draws on.