liveBook · Manning

Percentile

As explained in AI Model Evaluation

A latency percentile is a value below which a specified proportion of observed system response times falls.

A latency percentile is a response-time threshold below which a specified share of requests completes. Teams use p90 and p99 to distinguish typical performance from slow-tail behavior. section 4.4.2

Percentiles support practical performance decisions:

  • p99 means 99% of requests complete within the stated response time, while the slowest 1% may cause user-facing errors or timeouts. section 4.4.2
  • Use p90 and p99 to expose lagging requests and investigate bottlenecks in data retrieval, model inference, or system integrations. section 4.4.2
  • Treat percentiles as guardrail metrics during latency degradation experiments and monitor them alongside product, business, and error metrics. section 4.4.2 section 7.8.1
  • Preserve formula parity between offline calculations and production SQL or streaming pipelines so percentile comparisons remain trustworthy. section 7.8.1

A slow tail behind a healthy average

A backend service may appear responsive for most requests while its p99 remains high. The slowest 1% can exceed application thresholds, trigger fallbacks, and create timeouts; engineers then trace delays across retrieval, inference, and integrations. section 4.4.2

How AI Model Evaluation covers it

How deep it goes
Defines p90 and p99, explains tail latency and operational consequences, but does not detail percentile calculation algorithms.
Examples
Uses p90 and p99, backend response times, fallback states, latency experiments, and offline-to-production metric pipelines.
What it assumes
Assumes response-time and request concepts while establishing how percentile thresholds reveal uneven system performance.
What you can do after
Diagnose slow request tails, configure percentile guardrails, and validate that production metrics match offline computations.
How it connects
Connects latency percentiles with response times, tail latency, timeouts, fallbacks, bottlenecks, monitoring, and error rates.

Also explained in

Compare how 13 books explain percentile →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.