liveBook · Manning

Regression

As explained in Retrieval Augmented Generation, The Foundational Ideas

A regression is a deterioration in system behavior caused by a change that previously improved or preserved performance.

A regression is a deterioration in system behavior caused by a change that previously improved or preserved performance. In retrieval-augmented generation (RAG), changing the embedding model, index, chunking, retriever, reranker, generator, prompt, or evaluation data can alter the behavior of the whole pipeline. section 15.6.2

When better retrieval produces worse answers

A mortgage lender added a cross-encoder reranker and improved retrieval: returned documents answered questions more often. After deployment, faithfulness fell because answers drifted away from their supporting documents. The existing threshold did not expose the downward trend; comparing successive reports revealed the regression. section 14.0

Control regressions through a versioned, measurement-driven feedback loop:

  • Version the complete pipeline configuration rather than isolated components. section 15.6.2
  • Rebuild the index and rerun evaluation after embedding-model or corpus changes; test prompt changes before deployment. section 15.6.2
  • Compare current and previous reports, diagnose the changed failure point, apply a targeted intervention, then deploy and monitor. section 14.0 section 15.4.4
Listing 15.4 RAG pipeline version manifest.
from dataclasses import dataclass, field
from datetime import datetime
 
@dataclass
class RAGPipelineVersion:
    """Track RAG pipeline configuration for reproducibility."""
 
    version: str
    embedding_model: str
    chunk_size: int
    chunk_overlap: int
    retrieval_depth: int
    reranker: str | None
    generator_model: str
    system_prompt_hash: str  #A
    corpus_doc_count: int
    index_build_date: str  #B
    eval_scores: dict[str, float] = field(default_factory=dict)
 
    def to_dict(self) -> dict:  #C
        """Serialize for logging and comparison."""
        return {
            "version": self.version,
            "embedding": self.embedding_model,
            "chunk": f"{self.chunk_size}/{self.chunk_overlap}",
            "k": self.retrieval_depth,
            "reranker": self.reranker or "none",
            "generator": self.generator_model,
            "corpus_size": self.corpus_doc_count,
            "index_date": self.index_build_date,
            "scores": self.eval_scores,
        }
 
v1 = RAGPipelineVersion(
    version="1.2.0",
    embedding_model="text-embedding-3-small",
    chunk_size=512, chunk_overlap=50,
    retrieval_depth=5, reranker="bge-reranker-v2-m3",
    generator_model="gpt-4o-mini",
    system_prompt_hash="a3f2c1",
    corpus_doc_count=15420,
    index_build_date="2025-01-15",
    eval_scores={"faithfulness": 0.87, "relevance": 0.82},
)
print(v1.to_dict())  #D
 

A version manifest records the embedding model, chunk configuration, retrieval depth, reranker, generator, prompt hash, corpus size, index date, and evaluation scores, allowing configuration diffs when scores drop. section 15.6.2

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Explains interacting pipeline causes, trend detection, version diffs, targeted fixes, deployment, and monitoring.
Examples
Uses a mortgage compliance assistant, a cross-encoder reranker, successive evaluation reports, and a RAG pipeline version manifest.
What it assumes
Assumes familiarity with RAG components and evaluation metrics while emphasizing measurement discipline and reproducible configuration.
What you can do after
Diagnose regressions by comparing evaluation trends, diffing pipeline versions, and applying targeted configuration or model changes.
How it connects
Connects regression with RAG, retrieval, reranking, faithfulness, evaluation, prompt engineering, and continuous improvement.

Also explained in

Compare how 11 books explain regression →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.