liveBook · Manning

Baseline

As explained in Retrieval Augmented Generation, The Foundational Ideas

A baseline is a reference system or result against which another system's performance is compared.

A baseline is a reference system or result used to judge whether another system improves. In retrieval-augmented generation, metrics matter only beside a comparable reference: a baseline turns an isolated score into evidence about progress, tradeoffs, and failure points. section 3.2.1 section 15.5.3

Build and use a baseline before adding sophisticated techniques:

  • Choose a structurally similar reference system so training or knowledge-integration changes can be isolated. section 3.2.1
  • Evaluate on an unseen test split; memorized training examples can make an apparent improvement misleading. section 3.2.1
  • Record accuracy together with inference time and parameter count, because higher recall may cost latency or index memory. section 3.2.1
  • Establish RAGAS-triad measurements before advanced methods, then optimize the largest observed failure gap. section 15.5.3

A new RAG deployment

Start with naive RAG plus reranking for mostly factual queries. Measure retrieval and reader behavior from k=1 through k=20. If performance peaks and declines, improve reranking or context compression before increasing retrieval depth; if it improves and plateaus, retrieving more passages may be safe. section 15.5.3

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Defines comparison criteria, test-split discipline, and staged evaluation, but does not prescribe one universal baseline implementation.
Examples
Uses REALM and ORQA comparisons, unseen test splits, RAGAS measurements, and retrieval depths from k=1 through k=20.
What it assumes
Assumes familiarity with retrievers, reader models, retrieval depth, and evaluation metrics such as Recall@k and MRR.
What you can do after
Evaluate RAG configurations with comparable metrics, operational costs, unseen data, and failure-focused iteration.
How it connects
Connects retrieval evaluation to Recall@k, MRR, reranking, context compression, and reader-model noise sensitivity.

Also explained in

Compare how 27 books explain baseline →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.