A completion is a generated language-model output paired with a prompt. In preference fine-tuning and RLHF, it is the candidate action that may be scored, ranked, filtered, accepted, or used for further training. section 9.1.1 section 3.1.3
Treat completions as pipeline objects with several operational roles:
- Generate multiple candidates for each prompt, representing prompts as x and completions as y; for M prompts and N samples, each matrix row contains one prompt’s candidates. section 9.1.1
- Score a completion with a learned reward or preference model, often assigning one response-level reward to the full sequence rather than to individual tokens. section 3.1.3
- Use selected completions as instruction-tuning targets; the model generally learns from the completion tokens rather than predicting the prompt. section 4.2
- Compare a completion with the policy’s typical action through advantages, while constraining policy updates with a reference-model KL penalty during RLHF. section 6.1
Equation 9.2
\[\label{eq:rs_completions} Y = \begin{bmatrix} y_{1,1} & y_{1,2} & \cdots & y_{1,N} \\ y_{2,1} & y_{2,2} & \cdots & y_{2,N} \\ \vdots & \vdots & \ddots & \vdots \\ y_{M,1} & y_{M,2} & \cdots & y_{M,N} \end{bmatrix},\]A completion matrix makes candidate generation explicit: each row corresponds to a prompt, and each column contains one sampled candidate position across prompts. section 9.1.1
Candidate-completion pipeline
For prompts x₁ through x_M, generate yᵢ,₁ through yᵢ,ₙ for every prompt. A reward model scores the candidates, the pipeline retains the top completions, and instruction fine-tuning updates the original model on those selected outputs. section 9.1.1 section 9.0
