liveBook · Manning

Inference-time control

As explained in Retrieval Augmented Generation, The Foundational Ideas

Inference-time control is the adjustment of a model’s behavior during decoding by modifying how generated control signals affect generation.

Inference-time control adjusts a model’s behavior during decoding by changing how generated control signals influence candidate generation paths. Runtime token weights can favor strict factuality or permit more creativity without retraining. section 9.0

In a self-correcting generation loop, reflection signals guide this control: IsRel filters irrelevant documents, IsSup checks groundedness, and IsUse judges overall utility. section 9.0

Factuality versus creativity

An engineer can increase control-token weights when grounded, reliable answers matter, or relax them when more creative generation is acceptable, leaving model weights unchanged. section 9.0

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Explains reflection signals and runtime token weighting, but does not detail the decoding algorithm’s scoring procedure.
Examples
Uses IsRel, IsSup, and IsUse reflection tokens plus factuality-versus-creativity weighting.
What you can do after
Operate decoding by adjusting control-token weights without retraining the generator.
How it connects
Connects retrieval, self-critique, groundedness evaluation, decoding, and hallucination prevention.

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.