A span records one unit of work within a distributed request. It may represent a function execution, service invocation, database call, or other segment of processing. A trace groups spans into an end-to-end request, with a root span and nested child spans. section 8.1.2 section 8.4
A span typically carries its own ID, the trace ID, a parent or triggering span ID, timestamps, tags, and other metadata. Context propagation forwards these identifiers across HTTP, gRPC, or other network calls so separate services can be stitched together. section 2.5 section 8.1.2
Checkout request waterfall
A checkout request forms one trace. Its root span represents the checkout service, while child spans represent the billing service and a third-party payment-provider call. A waterfall makes the timing visible: if the payment span consumes most of the duration, it becomes the first optimization target. section 8.1.1
Spans support several operational workflows:
- Identify slow services and optimize request latency. section 2.5.4
- Compare execution times with a healthy baseline and locate the service responsible for an error. section 8.2.3 section 8.2.4
- Correlate traces with logs and metrics, or link causally related spans in asynchronous workloads. section 11.1.2
- Reduce collection volume through sampling, while accounting for instrumentation, transport, ingestion, and query costs. section 8.1.2 section 8.4
Tracing an injected failure
In Jaeger, filter spans by an HTTP status code or error=true. An error-marked span exposes the failing service, its duration, and its error code, allowing comparison with the happy-path trace and focused troubleshooting. section 8.2.4
Listing 2.10 Example Go program, instrumented to emit traces
func handleEcho(w http.ResponseWriter, r *http.Request) {
message := r.URL.Query().Get("message")
log.WithFields(log.Fields{
"service": "echo",
}).Info("Got input ", message)
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
tr := otel.Tracer("http-api")
_, span := tr.Start(ctx, "echo") #1
defer span.End()
if rand.Intn(100) > 90 {
log.WithFields(log.Fields{
"service": "echo",
}).Error("Something really bad happened :(")
invokes.WithLabelValues(
strconv.Itoa(http.StatusInternalServerError)).Inc()
span.SetAttributes(
attribute.Key("http-status-code").String(
strconv.Itoa(http.StatusInternalServerError))) #2
w.WriteHeader(500)
return
}
invokes.WithLabelValues(
strconv.Itoa(http.StatusOK)).Inc()
span.SetAttributes(
attribute.Key("http-status-code").String(
strconv.Itoa(http.StatusOK))) #3
fmt.Fprintf(w, message)
}
func main() {
rand.Seed(time.Now().UnixNano())
registry.MustRegister(invokes)
log.SetFormatter(&log.JSONFormatter{})
http.HandleFunc("/echo", handleEcho)
http.Handle("/metrics", promhttp.HandlerFor(
registry,
promhttp.HandlerOpts{
EnableOpenMetrics: true,
},
))
tp, err := tracerProvider("http://localhost:14268/api/traces") #4
if err != nil {
log.Fatal(err)
}
otel.SetTracerProvider(tp)
log.Fatal(http.ListenAndServe(":4242", nil))
}
A Go HTTP handler starts an OpenTelemetry span, ends it after processing, and records an HTTP status attribute when the operation fails. section 2.5.1
