liveBook · Manning

Adaptive retrieval

As explained in Retrieval Augmented Generation, The Foundational Ideas

Adaptive retrieval is a retrieval strategy that dynamically decides whether or when to retrieve information during generation.

Adaptive retrieval decides whether and when to fetch external information during generation instead of searching for every prompt. This avoids wasted compute, context pollution, and unnecessary exposure to irrelevant passages. section 9.0

A generator emits a Retrieve control token before or during generation. It can answer from parametric memory, request passages, repeat retrieval when a new sub-question appears, and stop when the available evidence is sufficient. section 9.0 section 9.3.3

Thresholding turns the retrieval decision into an adjustable production policy:

  • Compare P(Retrieve=Yes) with a threshold τ; skip the vector search when the probability is below τ. section 9.3.3
  • Lower τ to retrieve more often, improving recall on borderline knowledge questions while increasing compute and context-pollution risk. section 9.3.3
  • Raise τ to favor parametric answers and efficiency, accepting that some borderline questions may not trigger retrieval. section 9.3.3

One support assistant, three decisions

For “Thanks, that helped!”, P(Retrieve=Yes) is about 0.02, so the assistant answers without searching. For a v3-versus-v4 release-note question, probability rises to about 0.91; the assistant retrieves, then performs a second retrieval at about 0.78 for missing deprecation-policy details. For “Write a haiku about debugging,” probability is about 0.08, so retrieval is skipped. section 9.3.3

Adaptive retrieval can also pair retrieval with self-reflection: the model filters irrelevant documents, checks generated claims against relevant passages, and marks responses as supported or unverified through reflection tokens. section 9.2.1

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Explains Retrieve-token gating, repeated mid-generation searches, stopping criteria, and threshold tradeoffs through concrete probability examples.
Examples
Uses support acknowledgments, release-note comparisons, deprecation-policy follow-ups, and creative writing prompts.
What it assumes
Assumes familiarity with language generation, vector search, external passages, and the distinction between parametric and non-parametric memory.
What you can do after
Apply it via Retrieve-token probabilities and thresholds to skip, trigger, repeat, or stop retrieval during generation.
How it connects
Connects adaptive retrieval with self-reflection, reflection tokens, parametric memory, vector databases, and non-parametric memory.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.