liveBook · Manning

Retrieval depth

As explained in Retrieval Augmented Generation, The Foundational Ideas

Retrieval depth is the configured number or extent of candidate results considered during retrieval.

Retrieval depth is the number of candidate passages supplied to a reader at query time. Increasing it can raise the chance of finding relevant evidence, but extra context may add latency, token cost, and distracting noise. section 15.1.1

Reader behavior determines the useful range. Noise-resilient models such as FLAN-T5 and GPT-3.5 improve until roughly k=10 and then plateau. Noise-sensitive models such as LLaMA-3 and Claude-3-Haiku peak below k=5 and decline as distractors accumulate. section 15.1.1

Tune retrieval depth by measuring answer quality across candidate values rather than choosing a large k defensively:

  • Measure performance from k=1 through k=20 using the evaluation metric already tracked. section 15.1.2 section 15.5.3
  • Use the RAG Stability Score to detect brittleness around the best depth; values near 1 indicate stable performance. section 15.1.2
  • Use the RAG Scalability Coefficient to measure whether additional passages continue providing value, especially for multi-hop questions. section 15.1.2
  • Pair noise-sensitive readers with reranking and context compression before increasing retrieval depth; allow noise-robust readers to retrieve more safely. section 15.5.3

Robust and sensitive readers

A robust reader can remain near its peak across nearby depths, while a sensitive reader may peak at k=3 and deteriorate quickly as more passages arrive. The illustrative metric output gives RSS 0.945 and RSC 268.5 for the robust model, versus RSS 0.444 and RSC 70.5 for the sensitive model. section 15.1.2

Figure 15.1 Two distinct reader scaling behaviors from RAGGED.

Compare the plateauing performance curve of a noise-resilient reader with the early peak and decline of a noise-sensitive reader. section 15.1.1

Listing 15.1 Computing RAG stability score and scalability coefficient.
import numpy as np
 
def compute_rag_metrics(
    scores: list[float],
    delta: int = 2,
    epsilon: float = 0.5
) -> dict[str, float]:
    """Compute RSS and RSC from per-k performance scores."""
    scores_arr = np.array(scores, dtype=float)
    best = int(np.argmax(scores_arr))  #A
    peak = scores_arr[best]
    window_start = max(0, best - delta)
    window_end = min(len(scores_arr), best + delta + 1)
    window = np.concatenate([  #B
        scores_arr[window_start:best],
        scores_arr[best + 1:window_end]
    ])
    rss = float(np.min(window) / peak) if peak > 0 else 0.0
    k_last_gain = 0  #C
    for i in range(1, len(scores_arr)):
        if scores_arr[i] - scores_arr[i - 1] >= epsilon:
            k_last_gain = i
    rsc = 0.0
    for i in range(k_last_gain):
        rsc += 0.5 * (scores_arr[i] + scores_arr[i + 1])
    return {
        "k_star": best + 1,
        "rss": round(rss, 3),
        "rsc": round(rsc, 1),
    }
 
robust_model = [20, 35, 42, 48, 52, 54, 55, 55, 55, 55]  #D
sensitive_model = [20, 38, 45, 42, 38, 35, 33, 31, 30, 29]
 
print("Robust model:", compute_rag_metrics(robust_model))
print("Sensitive model:", compute_rag_metrics(sensitive_model))
 

Inspect the Python calculation of the optimal depth, stability score, and scalability coefficient from per-depth performance scores. section 15.1.2

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Explains reader scaling behaviors, evaluation metrics, and selection guidance, but stops short of retrieval-index internals.
Examples
Compares FLAN-T5, GPT-3.5, LLaMA-3, and Claude-3-Haiku across plateauing and declining depth curves.
What it assumes
Assumes familiarity with RAG readers and retrieval, while defining depth, stability, and scalability from measured performance.
What you can do after
Evaluate retrieval depth with per-k scores, RSS, and RSC, then select a reader-appropriate operating range.
How it connects
Connects retrieval depth with RAG, reranking, context compression, multi-step retrieval, and RAGAS evaluation.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.