liveBook · Manning

Reinforcement learning

As explained in Retrieval Augmented Generation, The Foundational Ideas

Reinforcement learning is a machine-learning paradigm in which an agent learns a policy from reward signals produced by interactions with an environment.

Reinforcement learning trains a policy from reward signals produced by interaction with an environment. In retrieval-augmented systems, it replaces per-document likelihood or human compression labels with task-level signals such as answer quality. section 6.6.2 section 13.4.1

CORE illustrates the workflow for a compressor: distill high-quality summaries first, then refine the policy with the downstream language model's exact-match and F1 scores. Group Relative Policy Optimization samples candidate summaries, compares their normalized group rewards, and avoids a separate critic model. section 13.4.1 section 13.4.1

The approach appears in several retrieval and compression decisions:

  • Train a search policy against the final answer rather than optimizing per-document likelihood. section 6.6.2
  • Train a compressor to optimize downstream task performance, reaching a three percent compression ratio while improving exact-match scores over an uncompressed baseline. section 13.4.1
  • Use reinforcement learning when generic compression quality or token pruning misses the facts required for multi-hop synthesis. section 13.4.1

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Explains two-phase training, candidate grouping, normalized advantages, and critic-free optimization, but does not develop general RL theory.
Examples
Uses Search-R1, ReSearch, CORE, RECOMP, and GRPO as examples of answer-level search or task-oriented compression.
What it assumes
Assumes a downstream language model and measurable task rewards, while showing how distillation can initialize a policy without annotated compression labels.
What you can do after
Apply it via policy optimization to align retrieval or compression with answer accuracy and task-specific evaluation.
How it connects
Connects reinforcement learning to retrieval augmentation, policy learning, language-model environments, knowledge distillation, and multi-hop reasoning.

Also explained in

Compare how 28 books explain reinforcement learning →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.