A retrieval relevance distribution summarizes the relevance scores attached to search results across many queries. Treating scores as a distribution, rather than a single average, exposes whether a system consistently retrieves useful context or hides a substantial low-quality tail. section 7.4.2
Use the distribution to make retrieval behavior observable and actionable:
- Record each query's relevance score as a histogram data point. section 7.4.2
- Attach dimensions such as provider, model, and workflow ID so operators can compare retrieval behavior across system contexts. section 7.4.2 section 7.5.2
- Inspect percentiles and low-quality portions instead of trusting an average that can conceal poor results for many queries. section 7.4.2
Production monitoring
A Data Service publishes relevance scores alongside search latency. A dashboard then displays retrieval relevance distributions, while a quality rule can read the score already attached to each search span. Operators can compare knowledge-base indexes and identify categories of questions that repeatedly receive weak retrieval results. section 10.10 section 7.5.2
Listing 7.4 Platform metric definitions and naming conventions
class PlatformMetrics:
"""Standard metrics emitted by platform services.
Names follow: ai.platform.{service}.{metric_name}"""
# Model Service
MODEL_REQUEST_TOTAL = \
"ai.platform.models.requests_total"
MODEL_REQUEST_DURATION = \
"ai.platform.models.request_duration_ms"
MODEL_TOKENS_PROMPT = \
"ai.platform.models.tokens.prompt"
MODEL_TOKENS_COMPLETION = \
"ai.platform.models.tokens.completion"
MODEL_COST_USD = \
"ai.platform.models.cost_usd"
MODEL_CACHE_HITS = \
"ai.platform.models.cache_hits_total" #A
MODEL_FALLBACKS = \
"ai.platform.models.fallbacks_total"
# Data Service
DATA_SEARCH_DURATION = \
"ai.platform.data.search_duration_ms"
DATA_RELEVANCE_SCORE = \
"ai.platform.data.relevance_score" #B
# Session Service
SESSION_CONTEXT_TOKENS = \
"ai.platform.sessions.context_tokens"
SESSION_TRUNCATIONS = \
"ai.platform.sessions.truncations_total"
# Guardrails Service
GUARDRAIL_VIOLATIONS = \
"ai.platform.guardrails.violations_total" #C
GUARDRAIL_LATENCY = \
"ai.platform.guardrails.evaluation_duration_ms"
# Tool Service
TOOL_EXECUTIONS = \
"ai.platform.tools.executions_total"
TOOL_CIRCUIT_BREAKS = \
"ai.platform.tools.circuit_breaks_total"Inspect the standard Data Service relevance metric and the naming convention used to publish it. section 7.4.2
















