liveBook · Manning

Critic-to-generator distillation

As explained in Retrieval Augmented Generation, The Foundational Ideas

Critic-to-generator distillation is a training technique that transfers critique behavior from a stronger critic model into a generator through supervised traces.

Critic-to-generator distillation addresses a reliability gap in retrieval-augmented generation: retrieval can add noise, while generation can remain unsupported or hallucinated. It transfers a stronger critic’s evaluation behavior into a smaller generator through supervised traces. section 9.0

A practical distillation pipeline follows these steps:

  • Ask a frontier critic to annotate query-answer examples with reflection tokens. section 9.0
  • Fine-tune the smaller generator on those annotated traces. section 9.0
  • Run the generator independently so it emits retrieval and quality-control signals during generation. section 9.0

Reflection-token transfer

A GPT-4 critic annotates a standard query-answer corpus with signals such as Retrieve, IsRel, IsSup, and IsUse. A Llama-2 7B or 13B generator learns to emit comparable decisions and critiques without calling the critic at inference time. section 9.0

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Explains critic annotation, generator fine-tuning, and token-guided decoding, but does not detail the training objective or dataset construction.
Examples
Uses GPT-4 annotations, Llama-2 7B/13B generators, query-answer traces, and Retrieve, IsRel, IsSup, and IsUse tokens.
What it assumes
Assumes a query-answer corpus, a stronger critic model, and familiarity with retrieval and grounded answer evaluation.
What you can do after
Build a smaller generator that reproduces a stronger critic’s retrieval and quality judgments during inference.
How it connects
Connects retrieval-augmented generation with self-critique, reflection tokens, groundedness, and supervised fine-tuning.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.