liveBook · Manning

Trace_generation

As explained in Designing AI Systems

trace_generation is a TracedService context-manager operation that creates and manages a generation trace primitive for a language-model call.

trace_generation is a TracedService context manager for instrumenting language-model calls. It creates a generation trace primitive, records model identity, timing, errors, token counts, cost, and time to first token, then yields a child context for nested operations. section 7.5.1

The operation keeps tracing off the request's critical path. start_generation() and end_generation() write locally to an in-memory ObservabilityClient buffer; later batching flushes completed generations through RecordGeneration. section 7.5.1

A typical generation-tracing flow:

  • Enter trace_generation with the incoming TraceContext and model name. section 7.5.1
  • Close the context so timing and errors are finalized while preserving parent-child trace relationships. section 7.5.1

ModelService.chat

A ModelService method wraps provider execution with `with self.trace_generation(trace_context, model=request.model) as gen_ctx:`. After receiving the response, it calls `gen_ctx.update(...)` with token usage and calculated cost before returning the response.

How Designing AI Systems covers it

How deep it goes
Explains context management, propagation, usage updates, buffering, and delivery, but defers buffering details.
Examples
Uses ModelService.chat with provider selection, response usage fields, calculated cost, and a generation-context update.
What it assumes
Assumes familiarity with platform services, context managers, TraceContext, provider responses, and observability clients.
What you can do after
Operate automatic generation tracing by wrapping provider calls and updating the yielded context with response metrics.
How it connects
Connects distributed tracing, TraceContext, ObservabilityClient, model services, token metrics, cost, and batched gRPC delivery.

This explainer was generated by Manning's knowledge graph from the text of Designing AI Systems. The section links point to the parts of the book it draws on.