liveBook · Manning

Rag scalability coefficient

As explained in Retrieval Augmented Generation, The Foundational Ideas

The RAG Scalability Coefficient measures how much additional retrieved context continues to improve model performance across retrieval depths.

The RAG Scalability Coefficient (RSC) measures whether a model keeps extracting value as retrieval depth increases. It accumulates the area under the performance curve until the last retrieval depth that still produces a gain of at least epsilon. Higher values indicate useful context integration across a wider range of k, especially for multi-hop questions. section 15.1.2

Use RSC alongside the RAG Stability Score to separate two retrieval-depth behaviors:

  • Peak-then-decline models benefit from a narrow retrieval range and may lose answer quality when additional passages introduce noise. section 15.9
  • Improving-and-plateauing models continue gaining value across more retrieved passages, producing a higher RSC. section 15.1.2
  • Evaluate RSC from per-depth answer-quality scores such as F1, exact match, or RAGAS faithfulness. section 15.1.2

Robust versus sensitive readers

A robust reader reaches its optimum at k=7, remains close to that peak, and scores RSC 268.5. A sensitive reader peaks at k=3, drops sharply around its optimum, and scores RSC 70.5. The RSC difference indicates how broadly each model benefits from additional retrieved context; RSS separately captures local brittleness. section 15.1.2

Listing 15.1 Computing RAG stability score and scalability coefficient.
import numpy as np
 
def compute_rag_metrics(
    scores: list[float],
    delta: int = 2,
    epsilon: float = 0.5
) -> dict[str, float]:
    """Compute RSS and RSC from per-k performance scores."""
    scores_arr = np.array(scores, dtype=float)
    best = int(np.argmax(scores_arr))  #A
    peak = scores_arr[best]
    window_start = max(0, best - delta)
    window_end = min(len(scores_arr), best + delta + 1)
    window = np.concatenate([  #B
        scores_arr[window_start:best],
        scores_arr[best + 1:window_end]
    ])
    rss = float(np.min(window) / peak) if peak > 0 else 0.0
    k_last_gain = 0  #C
    for i in range(1, len(scores_arr)):
        if scores_arr[i] - scores_arr[i - 1] >= epsilon:
            k_last_gain = i
    rsc = 0.0
    for i in range(k_last_gain):
        rsc += 0.5 * (scores_arr[i] + scores_arr[i + 1])
    return {
        "k_star": best + 1,
        "rss": round(rss, 3),
        "rsc": round(rsc, 1),
    }
 
robust_model = [20, 35, 42, 48, 52, 54, 55, 55, 55, 55]  #D
sensitive_model = [20, 38, 45, 42, 38, 35, 33, 31, 30, 29]
 
print("Robust model:", compute_rag_metrics(robust_model))
print("Sensitive model:", compute_rag_metrics(sensitive_model))
 

Inspect the Python listing to compute RSC and RSS from a sequence of per-retrieval-depth performance scores, with configurable delta and epsilon parameters. section 15.1.2

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Defines an area-under-curve measure through the last epsilon-sized gain, but does not specify a broader optimization procedure.
Examples
Compares robust and sensitive readers using per-depth scores, optimal k values, and reported RSC outputs.
What it assumes
Assumes retrieval-depth measurements and an existing answer-quality metric, while leaving the choice among F1, exact match, and RAGAS faithfulness open.
What you can do after
Apply it via Python to compare reader models and tune retrieval depth from measured answer-quality scores.
How it connects
Connects retrieval depth, multi-hop questions, RAG Stability Score, and adaptive retrieval.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.