A span records one timed unit of work within a distributed trace. When a request crosses several services, spans reveal which operations ran, how long they took, and whether they succeeded. Parent-child relationships organize the operations into a tree; a waterfall view makes their timing and bottlenecks visible. section 7.2.1 section 7.4.3
A span carries a trace_id shared by the request, its own span_id, and a parent_span_id identifying its caller. It also records the service and operation, start and end times, status, error message, attributes, and events. The root span represents the request; child spans describe nested work. Trace context travels between services in gRPC metadata, allowing the observability service to assemble the trace after individual services record their spans. section 7.4.3 section 7.3.1
Listing 7.5 Span, Generation, and Trace Protocol Buffer definitions
message Span {
string trace_id = 1; #A
string span_id = 2;
string parent_span_id = 3;
string service = 4;
string operation = 5; #B
google.protobuf.Timestamp start_time = 6;
google.protobuf.Timestamp end_time = 7;
SpanStatus status = 8;
string error_message = 9;
map<string, string> attributes = 10;
map<string, double> numeric_attributes = 11;
repeated SpanEvent events = 12;
}
message Generation {
Span span = 1; #C
string model = 2;
string provider = 3;
string requested_model = 4;
int32 prompt_tokens = 5;
int32 completion_tokens = 6;
double cost_usd = 7;
bool cache_hit = 8;
bool fallback_used = 9;
double time_to_first_token_ms = 10;
}
message Trace {
string trace_id = 1;
string session_id = 2; #D
repeated Span spans = 3;
repeated Generation generations = 4;
string input = 5;
string output = 6;
double total_duration_ms = 7;
double total_cost_usd = 8;
int32 total_tokens = 9;
repeated Score scores = 10; #E
repeated string tags = 11;
}Compare the Span fields with Generation, which wraps a span and adds model-specific fields. section 7.4.3
A generation specializes a span for an LLM call. It retains the span's timing and status while adding fields such as model, token counts, cost, cache status, and time to first token. Separate ingestion and indexing make those fields directly queryable without scanning generic spans. section 7.4.3
Finding a latency bottleneck
In a 2,370 ms patient intake request, a model generation takes 1,650 ms. The waterfall also shows retrieval, validation, and persistence spans, plus gaps representing communication overhead. Start with the trace to find the slow operation, inspect its logs using the trace_id, then check metrics to determine whether the problem is a pattern or an outlier. section 7.4.3
Figure 7.6 Distributed trace timeline for a patient intake request. The full request completes in 2,370 ms. The model generation step is the longest span at 1,650 ms, with associated metadata including model name, prompt tokens, completion tokens, and estimated cost. The remaining spans cover retrieval, validation, and persistence. Small gaps between spans represent communication overhead between services and help reveal where end-to-end latency is introduced

Inspect the request timeline and compare the generation's duration with other operations and communication gaps. section 7.4.3
Instrumentation can be automatic for platform service calls. The TracedService trace_operation context manager starts a span, records timing and errors, and yields a child context for downstream calls. The ObservabilityClient finalizes and buffers completed spans, then flushes batches through RecordSpan. Failed batches are retried, but the oldest buffered data is dropped if the buffer exceeds its limit. section 7.5.1 section 7.5.3
Application-specific work outside platform services needs explicit instrumentation. Wrap a custom reranking step, business rule, or external call with the SDK's trace_operation context manager to give it a span in the same trace tree instead of leaving a gap between automatically recorded operations. section 7.5.4
Listing 7.10 Adding a custom span for application-specific logic
with platform.observability.trace_operation(
"custom_rerank", trace_context, #A
num_candidates=len(candidates),
rerank_model="cross-encoder",
) as rerank_ctx:
reranked = my_reranker.rerank(candidates, query)See a custom reranking span with operation-specific attributes. section 7.5.4








