liveBook · Manning

Relevance

As explained in Designing AI Systems

A retrieval relevance distribution is the statistical distribution of relevance scores produced for retrieved results across queries.

A retrieval relevance distribution summarizes the relevance scores attached to search results across many queries. Treating scores as a distribution, rather than a single average, exposes whether a system consistently retrieves useful context or hides a substantial low-quality tail. section 7.4.2

Use the distribution to make retrieval behavior observable and actionable:

  • Record each query's relevance score as a histogram data point. section 7.4.2
  • Attach dimensions such as provider, model, and workflow ID so operators can compare retrieval behavior across system contexts. section 7.4.2 section 7.5.2
  • Inspect percentiles and low-quality portions instead of trusting an average that can conceal poor results for many queries. section 7.4.2

Production monitoring

A Data Service publishes relevance scores alongside search latency. A dashboard then displays retrieval relevance distributions, while a quality rule can read the score already attached to each search span. Operators can compare knowledge-base indexes and identify categories of questions that repeatedly receive weak retrieval results. section 10.10 section 7.5.2

Listing 7.4 Platform metric definitions and naming conventions
class PlatformMetrics:
    """Standard metrics emitted by platform services.
    Names follow: ai.platform.{service}.{metric_name}"""
 
    # Model Service
    MODEL_REQUEST_TOTAL = \
        "ai.platform.models.requests_total"
    MODEL_REQUEST_DURATION = \
        "ai.platform.models.request_duration_ms"
    MODEL_TOKENS_PROMPT = \
        "ai.platform.models.tokens.prompt"
    MODEL_TOKENS_COMPLETION = \
        "ai.platform.models.tokens.completion"
    MODEL_COST_USD = \
        "ai.platform.models.cost_usd"
    MODEL_CACHE_HITS = \
        "ai.platform.models.cache_hits_total"    #A
    MODEL_FALLBACKS = \
        "ai.platform.models.fallbacks_total"
 
    # Data Service
    DATA_SEARCH_DURATION = \
        "ai.platform.data.search_duration_ms"
    DATA_RELEVANCE_SCORE = \
        "ai.platform.data.relevance_score"    #B
 
    # Session Service
    SESSION_CONTEXT_TOKENS = \
        "ai.platform.sessions.context_tokens"
    SESSION_TRUNCATIONS = \
        "ai.platform.sessions.truncations_total"
 
    # Guardrails Service
    GUARDRAIL_VIOLATIONS = \
        "ai.platform.guardrails.violations_total"    #C
    GUARDRAIL_LATENCY = \
        "ai.platform.guardrails.evaluation_duration_ms"
 
    # Tool Service
    TOOL_EXECUTIONS = \
        "ai.platform.tools.executions_total"
    TOOL_CIRCUIT_BREAKS = \
        "ai.platform.tools.circuit_breaks_total"

Inspect the standard Data Service relevance metric and the naming convention used to publish it. section 7.4.2

How Designing AI Systems covers it

How deep it goes
Explains histogram aggregation, dimensional labels, and the limits of averages, but stops short of defining the scoring algorithm.
Examples
Uses Data Service search metrics, relevance-score histograms, dashboard distributions, and trace-based automated scoring.
What it assumes
Assumes familiarity with retrieval scores, histograms, labels, traces, and operational metrics.
What you can do after
Diagnose retrieval degradation by comparing score distributions across workflows, indexes, and question categories.
How it connects
Connects retrieval relevance scores with histograms, observability, dashboards, quality scoring, and automated evaluation.

Also explained in

Compare how 17 books explain relevance →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Designing AI Systems. The section links point to the parts of the book it draws on.