Sampling selectively captures or discards telemetry records according to a decision policy. In distributed tracing, it reduces the volume of traces and spans sent for collection, ingestion, and storage. section 8.1.2 section 8.3.1
Common strategies differ in when they decide and what they can know:
- No sampling ingests every span, which can suit development, testing, tool evaluation, or volume-neutral pricing. section 8.3.1
- Head-based sampling decides at the request’s beginning, making implementation simpler and reducing network costs, but potentially missing downstream errors or anomalies. section 2.5.3 section 8.3.1
- Tail-based sampling waits until the request completes, improving selection of relevant traces but requiring more state, resource usage, and a completion decision around timeouts or stragglers. section 2.5.3 section 8.3.1
- Policies can use fixed percentages or trace characteristics such as errors, while log sampling can drop a heuristic percentage of events. section 8.1.2 section 2.3.3
A microservice request with an error
A request moves across several microservices. Head-based sampling decides at the root before downstream behavior is known and may discard the request; tail-based sampling can retain it after an error appears in a downstream span, provided the collector waits long enough to determine completion. section 8.1.2 section 8.3.1
Table 2.3 Overview of the costs of traces (view table figure)
| Lots of commercial offerings and some OSS (e.g., Tempo and Jaeger) |
Use the trace-cost overview to place sampling among instrumentation, agent, storage, indexing, and query costs. section 2.5.3
