A regression is a deterioration in system behavior caused by a change that previously improved or preserved performance. In retrieval-augmented generation (RAG), changing the embedding model, index, chunking, retriever, reranker, generator, prompt, or evaluation data can alter the behavior of the whole pipeline. section 15.6.2
When better retrieval produces worse answers
A mortgage lender added a cross-encoder reranker and improved retrieval: returned documents answered questions more often. After deployment, faithfulness fell because answers drifted away from their supporting documents. The existing threshold did not expose the downward trend; comparing successive reports revealed the regression. section 14.0
Control regressions through a versioned, measurement-driven feedback loop:
- Version the complete pipeline configuration rather than isolated components. section 15.6.2
- Rebuild the index and rerun evaluation after embedding-model or corpus changes; test prompt changes before deployment. section 15.6.2
- Compare current and previous reports, diagnose the changed failure point, apply a targeted intervention, then deploy and monitor. section 14.0 section 15.4.4
Listing 15.4 RAG pipeline version manifest.
from dataclasses import dataclass, field
from datetime import datetime
@dataclass
class RAGPipelineVersion:
"""Track RAG pipeline configuration for reproducibility."""
version: str
embedding_model: str
chunk_size: int
chunk_overlap: int
retrieval_depth: int
reranker: str | None
generator_model: str
system_prompt_hash: str #A
corpus_doc_count: int
index_build_date: str #B
eval_scores: dict[str, float] = field(default_factory=dict)
def to_dict(self) -> dict: #C
"""Serialize for logging and comparison."""
return {
"version": self.version,
"embedding": self.embedding_model,
"chunk": f"{self.chunk_size}/{self.chunk_overlap}",
"k": self.retrieval_depth,
"reranker": self.reranker or "none",
"generator": self.generator_model,
"corpus_size": self.corpus_doc_count,
"index_date": self.index_build_date,
"scores": self.eval_scores,
}
v1 = RAGPipelineVersion(
version="1.2.0",
embedding_model="text-embedding-3-small",
chunk_size=512, chunk_overlap=50,
retrieval_depth=5, reranker="bge-reranker-v2-m3",
generator_model="gpt-4o-mini",
system_prompt_hash="a3f2c1",
corpus_doc_count=15420,
index_build_date="2025-01-15",
eval_scores={"faithfulness": 0.87, "relevance": 0.82},
)
print(v1.to_dict()) #D
A version manifest records the embedding model, chunk configuration, retrieval depth, reranker, generator, prompt hash, corpus size, index date, and evaluation scores, allowing configuration diffs when scores drop. section 15.6.2










