liveBook · Manning

Retrieve token

As explained in Retrieval Augmented Generation, The Foundational Ideas

A retrieve token is a control token whose predicted occurrence signals that external information should be retrieved during generation.

Retrieving for every prompt wastes money and can add irrelevant context. In Self-RAG, the Retrieve token is a machine-readable decision about whether generation needs external evidence. The orchestration layer reads the decision and either calls the retriever or continues without a new search; the token is not shown to the user. section 9.8 section 9.2

Retrieve can be predicted at the start of an answer or while it is being generated:

  • Yes: call the retriever for evidence. section 9.2.2
  • No: skip retrieval when the model can answer from its own parameters. section 9.2.2
  • Continue: keep using evidence already in the context window without issuing another query. section 9.2.2

A retrieval gate compares the predicted probability of Retrieve=Yes with a threshold before firing the vector store. It runs at each generation segment, so a new sub-question can trigger another search. Lowering the threshold retrieves more often, improving recall on borderline questions but increasing compute and the risk of noisy context. Retrieved passages then face an IsRel relevance check; IsSup checks support for generated sentences, and IsUse judges the answer’s utility. section 9.3.3 section 9.2.2

A support assistant with a 0.5 threshold

“Thanks, that helped!” has P(Retrieve=Yes)≈0.02, so the assistant replies without searching. A question comparing v3 and v4 release notes has probability ≈0.91 and triggers retrieval; a later need for deprecation-policy details triggers a second search at ≈0.78. A debugging haiku request, at ≈0.08, also skips retrieval. section 9.3.3

Figure 9.2 The full Self-RAG lifecycle: training, decoding, and adaptive retrieval. Left: the training phase (9.3.1) routes a raw query-answer dataset through a Critic model that annotates each example with inline reflection tokens, and the resulting traces fine-tune the Student model. Middle: the inference phase (9.3.2) shows the trained Student emitting candidate continuations alongside reflection tokens, with a token-guided beam search scoring each candidate by log P(text) + Σ w_T log P(T) over the IsRel, IsSup, and IsUse tokens, with w_T exposed as runtime knobs. Right: the adaptive retrieval gate (9.3.3) compares the Retrieve-token probability to a threshold; queries above it trigger the vector store, queries below it are answered from parametric memory.

The lifecycle diagram locates the Retrieve probability gate after training and alongside inference-time decoding, distinguishing the search decision from the other reflection-token scores. section 9.3

In trained Self-RAG, critic-annotated traces fine-tune a model to emit these control signals. A prompted implementation can instead use separate LLM calls and an orchestration graph without changing model weights, at the cost of additional round-trips. Log Retrieve decisions in production, and check token judgments against external labels: self-critique can share the assumptions behind the model’s mistakes. section 9.3 section 9.4 section 9.2.3 section 9.8

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Defines Yes, No, and Continue, then works through probability-threshold decisions and repeated retrieval cycles; it does not provide a standalone retriever implementation.
Examples
Uses support queries, release-note comparison, and a debugging haiku to contrast skipped searches with two retrieval cycles.
What it assumes
Assumes familiarity with token vocabularies and retrieval, while defining the Retrieve values and their control-flow effects.
What you can do after
Operate/configure it by choosing a retrieval threshold, gating searches during generation, and monitoring emitted decisions.
How it connects
Connects adaptive retrieval to reflection tokens, vector search, passage relevance, groundedness checks, and orchestration.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.