Retrieving for every prompt wastes money and can add irrelevant context. In Self-RAG, the Retrieve token is a machine-readable decision about whether generation needs external evidence. The orchestration layer reads the decision and either calls the retriever or continues without a new search; the token is not shown to the user. section 9.8 section 9.2
Retrieve can be predicted at the start of an answer or while it is being generated:
- Yes: call the retriever for evidence. section 9.2.2
- No: skip retrieval when the model can answer from its own parameters. section 9.2.2
- Continue: keep using evidence already in the context window without issuing another query. section 9.2.2
A retrieval gate compares the predicted probability of Retrieve=Yes with a threshold before firing the vector store. It runs at each generation segment, so a new sub-question can trigger another search. Lowering the threshold retrieves more often, improving recall on borderline questions but increasing compute and the risk of noisy context. Retrieved passages then face an IsRel relevance check; IsSup checks support for generated sentences, and IsUse judges the answer’s utility. section 9.3.3 section 9.2.2
A support assistant with a 0.5 threshold
“Thanks, that helped!” has P(Retrieve=Yes)≈0.02, so the assistant replies without searching. A question comparing v3 and v4 release notes has probability ≈0.91 and triggers retrieval; a later need for deprecation-policy details triggers a second search at ≈0.78. A debugging haiku request, at ≈0.08, also skips retrieval. section 9.3.3
Figure 9.2 The full Self-RAG lifecycle: training, decoding, and adaptive retrieval. Left: the training phase (9.3.1) routes a raw query-answer dataset through a Critic model that annotates each example with inline reflection tokens, and the resulting traces fine-tune the Student model. Middle: the inference phase (9.3.2) shows the trained Student emitting candidate continuations alongside reflection tokens, with a token-guided beam search scoring each candidate by log P(text) + Σ w_T log P(T) over the IsRel, IsSup, and IsUse tokens, with w_T exposed as runtime knobs. Right: the adaptive retrieval gate (9.3.3) compares the Retrieve-token probability to a threshold; queries above it trigger the vector store, queries below it are answered from parametric memory.

The lifecycle diagram locates the Retrieve probability gate after training and alongside inference-time decoding, distinguishing the search decision from the other reflection-token scores. section 9.3
In trained Self-RAG, critic-annotated traces fine-tune a model to emit these control signals. A prompted implementation can instead use separate LLM calls and an orchestration graph without changing model weights, at the cost of additional round-trips. Log Retrieve decisions in production, and check token judgments against external labels: self-critique can share the assumptions behind the model’s mistakes. section 9.3 section 9.4 section 9.2.3 section 9.8
