Generation produces text by repeatedly using a fixed, pretrained language model to predict the next token. Each new token extends the sequence and becomes context for the next prediction. section 2.1.1 section 2.3.1
A practical generation loop must handle more than selecting tokens:
- Tokenize prompts, call the policy model's generate function with sampling enabled, and record generated tokens. section 7.10
- Store old-policy log probabilities for generated tokens when completions will support PPO or GRPO updates. section 7.10
- Batch variable-length responses with padding and masks so early termination and padding do not affect scoring or gradients. section 7.10
Naive autoregressive generation repeatedly processes the entire growing sequence, making attention work quadratic in sequence length. A KV cache avoids recomputing prior key and value states, but shifts pressure toward GPU memory and bandwidth. section 2.3.1 section 2.5
Extending a short prompt
Given the prompt "The next day," generation processes the prompt and produces "is"; it then processes the extended sequence "The next day is" and produces "bright," followed by "and." section 2.3.1
|
Toy function
|
Real trainer equivalent
|
|
sample_group
|
Tokenizer + policy.generate + storage of sampled tokens
and old log probabilities
|
|
verify_math_answer
|
Task-specific verifier, unit tests, compiler, symbolic
checker, or reward model
|
|
group_advantages
|
Prompt-group reward normalization with numerical
safeguards
|
|
sequence_logprobs
|
Forward pass that gathers token log probabilities under
policy/reference models
|
|
grpo_loss
|
Masked clipped surrogate objective with KL control and
optimizer integration
|
Maps a toy sampling function to a real trainer's tokenizer, policy.generate call, sampled-token storage, and old-log-probability recording. section 7.10




























