liveBook · Manning

Self-rag

As explained in Retrieval Augmented Generation, The Foundational Ideas

SKILL-RAG is a retrieval-augmented generation technique that filters retrieved information according to the model's self-knowledge.

Retrieving documents for every query wastes compute and can fill the context with irrelevant passages. Self-RAG replaces that fixed sequence with an adaptive loop: a language model decides whether it needs retrieval, judges the passages it receives, and checks its answer against them before returning it. Unlike a conventional retrieve-then-generate pipeline, it can skip a search or retrieve again during generation. section 9.0 section 9.1.2

Four reflection tokens make those decisions explicit:

  • Retrieve chooses whether to search, skip searching, or continue using evidence already in context. section 9.2.2
  • IsRel marks each retrieved passage relevant or irrelevant, filtering passages that match a query without helping answer it. section 9.2.2
  • IsSup grades each generated sentence as fully supported, partially supported, or unsupported by the passage; unsupported claims can be revised or dropped. section 9.2.2
  • IsUse scores the completed answer’s usefulness, catching responses that are grounded but do not address the question. section 9.2.2
Table 9.1 The four Self-RAG reflection tokens
Token
Input
Output values
What the model is asserting
Retrieve
query (and partial generation)
Yes, No, Continue
Whether to call the retriever, skip it, or keep using evidence already in context
IsRel
query, retrieved passage
Relevant, Irrelevant
Whether the passage carries useful information for answering the query
IsSup
query, passage, generated sentence
Fully supported, Partially supported, No support
Whether every verification-worthy claim in the sentence is backed by the passage
IsUse
query, full generation
1, 2, 3, 4, 5
How useful the final response is as an answer to the query

Compare each token’s input and possible output before deciding where retrieval, passage filtering, claim checking, and answer scoring belong in a workflow. section 9.2.2

In trained Self-RAG, the model emits reflection tokens alongside text. During decoding, their probabilities help rank candidate continuations: relevant, supported, useful segments gain weight, while unsupported ones lose it. Engineers can change those weights at inference time to favor stricter grounding without retraining. This is an internal critique, not an independent fact-check: the model judging a claim can share the assumptions that produced it. section 9.3.2 section 9.8

Checking a compliance claim

Suppose a search retrieves an outdated regulation alongside a current one. A relevance check can exclude the historical passage. After drafting a compliance rating, a support check compares the claim with the current rule; if an internal policy requires quarterly audits but the rule requires monthly audits, the system rejects an unsupported “Compliant” answer. section 9.5

The original approach requires fine-tuning a model with annotated reflection tokens and a compatible tokenizer. Prompted Self-RAG instead uses an orchestration graph and separate LLM grader calls to imitate the decisions without changing model weights. It is easier to pilot but adds an API round-trip for each critique. Reserve the heavier loop for tasks where an incorrect answer is costly; cheaper options include a single-pass grounding instruction or an external evidence-alignment check. section 9.4 section 9.6

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Explains four token decisions and token-weighted decoding, then contrasts fine-tuning with a prompted orchestration graph.
Examples
Uses a Python-function prompt that needs no search and a regulatory audit where outdated evidence could support a false compliance rating.
What it assumes
Assumes familiarity with language models and RAG, while explaining reflection tokens and the difference between trained and prompted implementations.
What you can do after
Apply it via a prompted orchestration graph, or evaluate whether fine-tuning and added critique latency suit a high-trust deployment.
How it connects
Links adaptive retrieval to passage relevance, groundedness, beam search, prompted orchestration, and external fact-checking.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.