A histogram records observations as a distribution rather than collapsing them into an average. For AI services, it exposes latency and quality percentiles such as p50, p95, and p99, including heavy-tailed behavior that averages conceal. section 7.4.2 section 7.5.2
Use a histogram when:
- Latency varies widely and operators need percentile views instead of a single average. section 7.4.2
- Quality values such as retrieval relevance contain a problematic low-scoring fraction hidden by the mean. section 7.4.2
- Each request can contribute one observation while labels preserve dimensions such as provider, model, workflow, and cache status. section 7.5.2
Publishing request latency
A service publisher sends MODEL_REQUEST_DURATION and the request's duration_ms to record_histogram, alongside counters for request totals and cost. Thousands of observations then support queries such as p95 latency for one model and workflow. section 7.5.2
Listing 7.8 Publishing metrics from the Model Service
class ModelServiceMetricsPublisher:
def __init__(self, observability: ObservabilityClient):
self.observability = observability
def publish(self, request_metrics:
➥RequestMetrics): #A
labels = {
"provider": request_metrics.provider,
"model": request_metrics.model,
"workflow_id":
request_metrics.workflow_id or "unknown",
"cache_hit": str(request_metrics.cache_hit),
}
self.observability.record_counter( #B
PlatformMetrics.MODEL_REQUEST_TOTAL, 1, labels)
self.observability.record_histogram(
PlatformMetrics.MODEL_REQUEST_DURATION,
request_metrics.duration_ms, labels)
self.observability.record_counter(
PlatformMetrics.MODEL_COST_USD,
self._calculate_cost(request_metrics), labels)Inspect the publisher interface that translates request data into labeled histogram and counter calls. section 7.5.2












