liveBook · Manning

Generation

As explained in Designing AI Systems

A generation is an observability record that wraps a span and adds model, provider, token, cost, cache, and first-token timing information for a model call.

A generation is a specialized span for a model call. It preserves the timing, status, attributes, and trace relationships of a span while adding model-specific fields. Keeping those fields first-class lets operators query model behavior directly instead of scanning generic spans for key-value attributes. section 7.2.1 section 7.4.3

Listing 7.5 Span, Generation, and Trace Protocol Buffer definitions
message Span {
  string trace_id = 1;                          #A
  string span_id = 2;
  string parent_span_id = 3;
  string service = 4;
  string operation = 5;                         #B
  google.protobuf.Timestamp start_time = 6;
  google.protobuf.Timestamp end_time = 7;
  SpanStatus status = 8;
  string error_message = 9;
  map<string, string> attributes = 10;
  map<string, double> numeric_attributes = 11;
  repeated SpanEvent events = 12;
}
 
message Generation {
  Span span = 1;                                #C
  string model = 2;
  string provider = 3;
  string requested_model = 4;
  int32 prompt_tokens = 5;
  int32 completion_tokens = 6;
  double cost_usd = 7;
  bool cache_hit = 8;
  bool fallback_used = 9;
  double time_to_first_token_ms = 10;
}
 
message Trace {
  string trace_id = 1;
  string session_id = 2;                        #D
  repeated Span spans = 3;
  repeated Generation generations = 4;
  string input = 5;
  string output = 6;
  double total_duration_ms = 7;
  double total_cost_usd = 8;
  int32 total_tokens = 9;
  repeated Score scores = 10;                   #E
  repeated string tags = 11;
}

The Generation message wraps a Span and adds the fields needed to track model usage and performance. section 7.4.3

The model-specific fields record:

  • The model, provider, and requested model. section 7.4.3
  • Prompt and completion token counts, plus cost in USD. section 7.4.3
  • Cache-hit and fallback status, plus time to first token in milliseconds. section 7.4.3

A generation belongs in the same trace tree as other work. Services propagate trace context, and the observability service correlates recorded spans and generations by trace_id to assemble the request. Scores can attach to a generation to capture quality signals separately from execution telemetry. section 7.4.3 section 7.3.1

A model call within a request

A patient intake request records a GPT-4o generation with 3,100 prompt tokens, 180 completion tokens, $0.018 cost, 340 ms to first token, and no cache hit. Retrieval, validation, and persistence are recorded as regular spans in the same trace. section 7.2.2

Instrument model calls with a generation context manager rather than the generic operation wrapper. It creates the record, captures timing and errors, and provides a child context for nesting. Local start and end methods buffer telemetry; completed generations are sent in batches through RecordGeneration. The separate ingestion path supports a generation-specific index for queries such as calls above a cost threshold. section 7.5.1 section 7.3.1

Listing 7.7 Model Service chat method with automatic generation tracking
class ModelService(TracedService):
 
    def chat(self, request, trace_context):
        with self.trace_generation(    #A
            trace_context, model=request.model,
        ) as gen_ctx:
            provider = self._select_provider
➥(request)    #B
            response = provider.chat(
                request.messages, request.config)
            gen_ctx.update(    #C
                prompt_tokens=response.usage.prompt_tokens,
                completion_tokens=response.usage.completion_tokens,
                cost_usd=self._calculate_cost(response),
            )
            return response

The chat implementation opens a generation context and updates it with token usage and calculated cost after the provider responds. section 7.5.1

How Designing AI Systems covers it

How deep it goes
Defines the generation schema and ingestion contract, then implements context-managed tracking and buffered delivery.
Examples
Traces a patient intake model call and instruments a Model Service chat method.
What you can do after
Build generation tracking with Python context managers and buffered gRPC ingestion.
How it connects
Connects distributed tracing, spans, sessions, quality scores, and observability metrics.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Designing AI Systems. The section links point to the parts of the book it draws on.