Retrieving documents for every query wastes compute and can fill the context with irrelevant passages. Self-RAG replaces that fixed sequence with an adaptive loop: a language model decides whether it needs retrieval, judges the passages it receives, and checks its answer against them before returning it. Unlike a conventional retrieve-then-generate pipeline, it can skip a search or retrieve again during generation. section 9.0 section 9.1.2
Four reflection tokens make those decisions explicit:
- Retrieve chooses whether to search, skip searching, or continue using evidence already in context. section 9.2.2
- IsRel marks each retrieved passage relevant or irrelevant, filtering passages that match a query without helping answer it. section 9.2.2
- IsSup grades each generated sentence as fully supported, partially supported, or unsupported by the passage; unsupported claims can be revised or dropped. section 9.2.2
- IsUse scores the completed answer’s usefulness, catching responses that are grounded but do not address the question. section 9.2.2
Table 9.1 The four Self-RAG reflection tokens
|
Token
|
Input
|
Output values
|
What the model is asserting
|
|
Retrieve
|
query (and partial generation)
|
Yes, No, Continue
|
Whether to call the retriever, skip it, or keep using
evidence already in context
|
|
IsRel
|
query, retrieved passage
|
Relevant, Irrelevant
|
Whether the passage carries useful information for
answering the query
|
|
IsSup
|
query, passage, generated sentence
|
Fully supported, Partially supported, No support
|
Whether every verification-worthy claim in the sentence is
backed by the passage
|
|
IsUse
|
query, full generation
|
1, 2, 3, 4, 5
|
How useful the final response is as an answer to the query
|
Compare each token’s input and possible output before deciding where retrieval, passage filtering, claim checking, and answer scoring belong in a workflow. section 9.2.2
In trained Self-RAG, the model emits reflection tokens alongside text. During decoding, their probabilities help rank candidate continuations: relevant, supported, useful segments gain weight, while unsupported ones lose it. Engineers can change those weights at inference time to favor stricter grounding without retraining. This is an internal critique, not an independent fact-check: the model judging a claim can share the assumptions that produced it. section 9.3.2 section 9.8
Checking a compliance claim
Suppose a search retrieves an outdated regulation alongside a current one. A relevance check can exclude the historical passage. After drafting a compliance rating, a support check compares the claim with the current rule; if an internal policy requires quarterly audits but the rule requires monthly audits, the system rejects an unsupported “Compliant” answer. section 9.5
The original approach requires fine-tuning a model with annotated reflection tokens and a compatible tokenizer. Prompted Self-RAG instead uses an orchestration graph and separate LLM grader calls to imitate the decisions without changing model weights. It is easier to pilot but adds an API round-trip for each critique. Reserve the heavier loop for tasks where an incorrect answer is costly; cheaper options include a single-pass grounding instruction or an external evidence-alignment check. section 9.4 section 9.6
