liveBook · Manning

Rag stability score

As explained in Retrieval Augmented Generation, The Foundational Ideas

The RAG Stability Score measures how consistently retrieval-augmented generation performance remains near its optimal retrieval depth.

The RAG Stability Score (RSS) measures how consistently a reader model performs near its optimal retrieval depth. It matters when retrieval quality fluctuates and changing k can move a system away from its best configuration. section 15.1.1 section 15.1.2

Let k* be the retrieval depth with the highest answer-quality score. RSS divides the lowest score within a chosen window around k* by the peak score. Values near 1 indicate a forgiving configuration; lower values indicate brittleness. section 15.1.2

Use RSS to distinguish two retrieval-depth behaviors:

  • Improve-then-plateau readers gain quality until roughly k=10, then remain stable; FLAN-T5 reached RSS 0.99. section 15.1.1
  • Peak-then-decline readers perform best at small k and degrade with added distractors; LLaMA-3 scored about 0.63–0.64. section 15.1.1
  • Treat retrieval depth as a production tuning parameter for noise-sensitive readers rather than retrieving many passages defensively. section 15.1.1

Robust versus sensitive readers

Illustrative scores produced RSS 0.945 for a robust reader and 0.444 for a sensitive reader. The robust reader stayed within 6% of its peak around k*, while the sensitive reader varied sharply near its optimum. section 15.1.2

Figure 15.1 Two distinct reader scaling behaviors from RAGGED.

Compare plateauing and declining reader-performance curves as retrieval depth increases. section 15.1.1

Listing 15.1 Computing RAG stability score and scalability coefficient.
import numpy as np
 
def compute_rag_metrics(
    scores: list[float],
    delta: int = 2,
    epsilon: float = 0.5
) -> dict[str, float]:
    """Compute RSS and RSC from per-k performance scores."""
    scores_arr = np.array(scores, dtype=float)
    best = int(np.argmax(scores_arr))  #A
    peak = scores_arr[best]
    window_start = max(0, best - delta)
    window_end = min(len(scores_arr), best + delta + 1)
    window = np.concatenate([  #B
        scores_arr[window_start:best],
        scores_arr[best + 1:window_end]
    ])
    rss = float(np.min(window) / peak) if peak > 0 else 0.0
    k_last_gain = 0  #C
    for i in range(1, len(scores_arr)):
        if scores_arr[i] - scores_arr[i - 1] >= epsilon:
            k_last_gain = i
    rsc = 0.0
    for i in range(k_last_gain):
        rsc += 0.5 * (scores_arr[i] + scores_arr[i + 1])
    return {
        "k_star": best + 1,
        "rss": round(rss, 3),
        "rsc": round(rsc, 1),
    }
 
robust_model = [20, 35, 42, 48, 52, 54, 55, 55, 55, 55]  #D
sensitive_model = [20, 38, 45, 42, 38, 35, 33, 31, 30, 29]
 
print("Robust model:", compute_rag_metrics(robust_model))
print("Sensitive model:", compute_rag_metrics(sensitive_model))
 

Compute RSS from per-depth performance scores and compare it with the separate RAG Scalability Coefficient. section 15.1.2

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Defines a windowed peak-relative ratio and contrasts stable plateaus with sharp performance declines, without detailing retrieval internals.
Examples
Uses FLAN-T5, LLaMA-3, illustrative score lists, and performance curves across retrieval depths.
What it assumes
Assumes per-retrieval-depth answer-quality scores and teaches stability from the retrieval-depth optimum onward.
What you can do after
Evaluate reader robustness and select retrieval depths that avoid instability near the measured optimum.
How it connects
Connects retrieval depth, reader models, RAG evaluation, and the RAG Scalability Coefficient.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.