liveBook · Manning

Histogram

As explained in Designing AI Systems

A histogram metric records observations in value ranges so their distribution and percentile statistics can be calculated.

A histogram records observations as a distribution rather than collapsing them into an average. For AI services, it exposes latency and quality percentiles such as p50, p95, and p99, including heavy-tailed behavior that averages conceal. section 7.4.2 section 7.5.2

Use a histogram when:

  • Latency varies widely and operators need percentile views instead of a single average. section 7.4.2
  • Quality values such as retrieval relevance contain a problematic low-scoring fraction hidden by the mean. section 7.4.2
  • Each request can contribute one observation while labels preserve dimensions such as provider, model, workflow, and cache status. section 7.5.2

Publishing request latency

A service publisher sends MODEL_REQUEST_DURATION and the request's duration_ms to record_histogram, alongside counters for request totals and cost. Thousands of observations then support queries such as p95 latency for one model and workflow. section 7.5.2

Listing 7.8 Publishing metrics from the Model Service
class ModelServiceMetricsPublisher:
    def __init__(self, observability: ObservabilityClient):
        self.observability = observability
 
    def publish(self, request_metrics: 
➥RequestMetrics):    #A
        labels = {
            "provider": request_metrics.provider,
            "model": request_metrics.model,
            "workflow_id":
                request_metrics.workflow_id or "unknown",
            "cache_hit": str(request_metrics.cache_hit),
        }
        self.observability.record_counter(    #B
            PlatformMetrics.MODEL_REQUEST_TOTAL, 1, labels)
        self.observability.record_histogram(
            PlatformMetrics.MODEL_REQUEST_DURATION,
            request_metrics.duration_ms, labels)
        self.observability.record_counter(
            PlatformMetrics.MODEL_COST_USD,
            self._calculate_cost(request_metrics), labels)

Inspect the publisher interface that translates request data into labeled histogram and counter calls. section 7.5.2

How Designing AI Systems covers it

How deep it goes
Explains distribution rationale, percentile use, labels, and publishing calls, but defers storage and percentile calculation internals.
Examples
Uses model request duration and retrieval relevance as concrete histogram metrics, queried by provider, model, and workflow.
What it assumes
Assumes familiarity with metrics, service request data, labels, and counters while establishing why averages are insufficient.
What you can do after
Apply it via an ObservabilityClient to publish labeled observations and inspect p50, p95, and p99 behavior.
How it connects
Connects histograms with metrics, counters, dimensional labels, dashboards, and observability pipelines.

Also explained in

Compare how 13 books explain histogram →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Designing AI Systems. The section links point to the parts of the book it draws on.