Rollout refers to a complete answer generated by the model for a given prompt. In GRPO, multiple rollouts are sampled for each prompt, scored as full candidates, and compared against one another for policy updates. section 6.2.1 section 6.5 section 6.9
The treatment keeps rollout close to completion, noting that the two terms are often used interchangeably in the LLM setting. It emphasizes full-sequence generation rather than step-by-step judging or partial feedback. section 6.2.1 section 6.9
Rollouts are used here to support the GRPO workflow by:
- generating several candidate answers for one prompt. section 6.2.1
- providing the candidates that later receive reward and relative comparison. section 6.2.1
- feeding sequence-level logprobs into the policy gradient loss. section 6.9
Sampling a math answer
A math prompt is rendered, then sample_response generates a rollout with temperature and top-p sampling. The returned answer text, token IDs, and prompt length make the candidate usable in later GRPO stages. section 6.5 section 6.3
Figure 6.8 Overview of the GRPO algorithm for RLVR using a chef analogy. Multiple rollouts are generated and scored, relative advantages are computed, and a policy gradient objective with a KL-based regularization loss term is used to update the model parameters.
Shows multiple rollouts scored relative to each other before a policy-gradient update with KL regularization. section 6.2.1
