A latency percentile is the threshold below which a specified percentage of measured requests complete. The 95th-percentile latency contains 95% of requests, while the 99th and 99.9th percentiles expose progressively slower behavior. Percentiles describe a distribution rather than collapsing performance into one average; minimum and maximum show only its extremes. section 2.2 section 2.7
High percentiles are called tail latency. They matter because users encounter slow requests more often than an apparent outlier suggests, and parallel services can become bound by their slowest subrequest. Increasing fanout therefore increases the chance that service latency exceeds a threshold. section 2.2 section 2.7
Measure and inspect percentiles with a distribution-oriented workflow:
- Collect latency samples while defining what is measured and controlling the measurement environment. section 2.7
- Plot a histogram to reveal the main cluster and sparse right-hand tail. section 2.6.1
- Use HdrHistogram to obtain values at selected percentiles such as 25%, 50%, 90%, 99%, 99.9%, and 99.99%. section 2.6.1
- Compare tail percentiles when evaluating optimizations, because component variability, queuing, physical limits, operating-system configuration, and application design can shape latency. section 2.2 section 2.7
A fanout example
Suppose a service sends requests to multiple servers in parallel and waits for every response. Even when most subrequests finish in about ten milliseconds, an occasional slow subrequest determines the completed service request's latency. The relevant question is therefore not only the typical latency, but how the 95th, 99th, or higher percentile changes as parallel fanout grows. section 2.2
