A trace tree is a hierarchical view of a request across services. Parent and child spans show which operations called others, how long they took, and where the request stopped. section 12.1.1
Use a trace tree when logs cannot explain a failure spanning service boundaries. Follow the trace ID, inspect missing or incomplete spans, and treat a branch that ends before a database write or message publish as a diagnostic red flag. section 13.1.1 section 12.1.1
Read a trace tree by asking:
- Which service produced the last recorded span? section 13.1.1
- Does the tree include the expected database write, message publication, or downstream service call? section 13.1.1
- Do span durations, status fields, tags, logs, or parent references suggest where execution diverged? section 12.1.1
- Do Kafka inspection, application logs, and direct database queries confirm what the trace shows? section 13.1.1
A missing payment branch
A trace can show POS initiatePayment() calling an API gateway and payment-processing service, followed by no further spans. Investigate that last service, then compare the trace with Kafka records, application logs, database gaps, and PostgreSQL deadlock logs. A deadlock or unhandled transaction rollback may explain why the downstream action never completed. section 13.1.1
Listing 12.1 An example of a trace timeline also called a trace tree
Trace ID: 6f98c1e2b3a24f59
└── [api-gateway] POST /checkout [950ms]
├── [checkout-service] Validate cart [40ms]
├── [checkout-service] Call OrderService [780ms]
│ └── [order-service] Create order record [100ms]
│ ├── [order-service] Check inventory [60ms]
│ └── [order-service] Save to database [30ms]
├── [checkout-service] Call PaymentService [100ms]
│ └── [payment-service] Process payment [90ms]
└── [checkout-service] Send confirmation email [30ms]
Inspect a concrete checkout trace showing nested validation, service calls, database persistence, payment, and confirmation-email spans with durations. section 12.1.1
