liveBookmanning.com

What you will learn in Reinforcement Learning from Human Feedback

Alignment and post-training of LLMs

by Nathan Lambert
Foreword by Thomas Wolf

Read it on liveBook Get the book at manning.com

Chapter by chapter

Concepts explained in this book

• reward modelA reward model is a learned model that predicts the reward or preference value assigned to an output or behavior.
• reinforcement learning with human feedbackReinforcement learning from human feedback (RLHF) is a model-alignment technique that optimizes behavior using human preference signals, often through a learned reward model or direct preference objective.
• preference dataA pairwise comparison is an evaluation that determines which of two items is preferred.
• fine-tuningFine-tuning is further training of a pretrained model on a task-specific or behavior-specific objective.
• post-trainingPost-training is the set of optimization and adaptation procedures applied to a pretrained model to improve its usefulness for downstream tasks or alignment objectives.
• reinforcement learningReinforcement learning is a machine learning paradigm in which an agent learns behavior by optimizing rewards obtained through interaction with an environment.
• Kullback–Leibler divergenceForward KL divergence is KL divergence from a target distribution to an approximating distribution, with expectation taken under the target distribution.
• advantage functionAn advantage function estimates how much better an action is than the policy's expected value at a state.
• Proximal Policy OptimizationPPO is a policy-gradient reinforcement-learning algorithm that constrains policy updates to remain near a reference policy.
• Direct Preference OptimizationDirect preference optimization (DPO) directly trains a policy from pairwise preferred and dispreferred outputs relative to a reference policy.
• reinforcement learning with verifiable rewardsReinforcement learning with verifiable rewards (RLVR) is reinforcement learning in which rewards are derived from objectively checkable outcomes.
• policy gradient methodPolicy gradient is a reinforcement-learning technique that optimizes policy parameters by estimating gradients of expected reward from sampled trajectories.
• value functionA value network is a neural network that estimates the expected value of states or actions in reinforcement learning.
• preferenceA human preference is a person's comparative judgment about the desirability or quality of model behavior or outputs.
• over-optimizationReward over-optimization is a condition in which a reward model's score continues to improve while actual quality measured by held-out evaluations or human judgment degrades.
• KL regularizationA KL penalty is a regularization term proportional to the Kullback–Leibler divergence between an optimized distribution or policy and a reference distribution or policy.
• reward signalA reward is a scalar evaluative signal indicating the desirability of an action or outcome for an agent.
• importance weightA probability ratio is the quotient of the probability assigned to an outcome by a target policy or distribution and the probability assigned by a sampling or reference policy or distribution.
• on-policy methodOn-policy optimization, or on-policy learning, updates a policy using data generated by that same policy or its current update distribution.
• preference tuningPreference optimization is model training that adjusts behavior toward preferred outputs using preference-based objectives or rewards.
• reasoning modelA reasoning language model is a language model optimized to solve problems through explicit or implicit multi-step reasoning before producing a final answer.
• policyA policy is a function that maps an environment state to a probability distribution over actions.
• regularizationRegularization is a technique that constrains optimization to reduce undesirable or excessive fitting to an objective.
• output formatResponse format is the structural arrangement and presentation convention of a model response.
• Group Relative Policy OptimizationGRPO is a policy-optimization algorithm that estimates relative advantages among groups of sampled outputs.
• inference-time compute scalingInference-time scaling improves model performance by allocating additional computation during inference, such as generating and evaluating more candidate reasoning paths.
• evaluationModel evaluation is the systematic assessment of a model's behavior or performance against defined tasks, criteria, or benchmarks.
• negative log-likelihoodNegative log-likelihood is the negation of a model's log-likelihood, used as a loss for probabilistic predictions.
• synthetic preference dataSynthetic preference data is preference data generated by models or algorithmic processes rather than directly collected from human judgment.
• clipped surrogate objectiveA PPO surrogate objective, also called a clipped surrogate objective, is an importance-weighted policy-gradient objective whose probability ratio is clipped to limit policy updates.
• tool usageTool use is a model capability in which generated interactions invoke external tools or services to perform tasks.
• trajectoryAn episode is a finite sequence of interaction steps experienced by an agent during one trial.
• discount factorA discount factor is a scalar between zero and one that weights future rewards relative to nearer rewards.
• reverse KL divergenceReverse KL divergence is the Kullback-Leibler divergence from a target distribution to a reference distribution, penalizing probability mass assigned by the target where the reference assigns little probability.
• reasoning model trainingReasoning training is the optimization of a model to produce or support more effective multi-step problem-solving behavior.
• cross entropy lossCross-entropy is the expected negative log probability assigned by one distribution to outcomes drawn from another distribution.
• verifiable rewardA verifiable reward is a reward signal whose correctness can be checked objectively by a procedure or criterion.
• AI feedbackAI feedback is evaluative information about model outputs or actions generated by an artificial-intelligence system rather than directly supplied by people.
• verificationA verification function evaluates a candidate answer or behavior against an objectively checkable correctness criterion.
• direct alignment algorithmA direct alignment algorithm optimizes a model directly from preference data without learning an intermediate reward model.
• asynchronous trainingAsynchronous reinforcement learning is an RL training arrangement in which rollout generation and policy optimization proceed concurrently on separate workers with periodically synchronized parameters.
• outcome reward modelAn outcome reward model assigns a reward to the final outcome of a task or trajectory rather than to intermediate reasoning steps.
• importance samplingImportance sampling is a method for estimating an expectation under one probability distribution using samples drawn from another distribution and probability weights.
• distillationDistillation is a model-training technique in which a student model learns to reproduce the outputs or behavior of a teacher model.
• proxy objectiveA proxy objective is an optimization objective that approximates a desired goal instead of measuring that goal directly.
• RLOORLOO, or REINFORCE leave one out, is a policy-gradient algorithm that estimates each sample's advantage relative to the average reward of the other samples for the same prompt.
• contrastive loss functionA contrastive loss is an optimization objective that compares examples to encourage similar or preferred examples to be distinguished from dissimilar or non-preferred ones.
• foundational modelA foundation model is a broadly pretrained model that can be adapted to support multiple downstream tasks.
• catastrophic forgettingCatastrophic forgetting is the rapid loss of previously learned capabilities or information when a model is trained on new objectives or data.
• Bradley-Terry modelA Bradley–Terry model is a probabilistic model that represents pairwise preferences using latent positive strengths or equivalent real-valued scores for items.
• gradient descentA gradient update, or gradient step, changes model parameters using the gradient of an optimization objective.
• reference modelA reference model is a frozen copy of a policy that provides a baseline distribution for measuring or penalizing policy deviation.
• chat templateA chat template is a formatting specification that converts structured conversation messages into the token sequence expected by a language model.
• styleResponse style is the characteristic manner in which a model presents content, including presentation and rhetorical qualities such as bluntness or warmth.
• synthetic dataSynthetic data is data generated by an algorithmic or model-based process rather than directly collected from the target phenomenon.
• benchmarkA benchmark is a standardized set of questions, tasks, or environments designed to measure a specific property of a model.
• personality trainingCharacter training is model training aimed at precisely controlling a model's personality and behavioral character.
• proxy metricA proxy metric is an imperfect measurable quantity used to approximate a desired quality or objective.
• thinking tokenA thinking token is a generated language-model token belonging to an intermediate reasoning sequence produced before a final response.
• DAPODynamic sampling policy optimization (DAPO) is a GRPO variant that combines dynamic sampling, decoupled clipping, token-level optimization, and length-aware penalties for reasoning-model training.
• partition functionA partition function is a normalization factor that converts unnormalized nonnegative weights into a probability distribution.
• instruction dataAn instruction dataset is a collection of prompts or instructions paired with target responses for supervised model adaptation.
• DPO lossThe DPO loss is a differentiable preference-training objective that increases the relative likelihood of chosen completions over rejected completions relative to a reference policy.
• Markov decision processA Markov decision process is a mathematical framework that models states, actions, rewards, and probabilistic transitions for sequential decision making under the Markov property.
• reward-to-goReward-to-go is the cumulative future reward expected from a specified point in a trial or trajectory.
• evaluatorLLM-as-a-judge is an evaluation technique in which a language model assesses the quality or preference of model outputs according to provided criteria.
• REINFORCEREINFORCE is a policy-gradient reinforcement-learning algorithm that updates a policy using sampled returns or rewards.
• difficulty filteringDifficulty filtering is a training-data selection technique that retains or emphasizes examples according to their estimated difficulty.
• objective functionA loss function is a scalar objective that quantifies prediction or optimization error during model training.
• accuracyTraining accuracy is the proportion of training examples for which a model produces the correct output.
• log-ratioA log-ratio is the logarithm of the ratio between two positive quantities, such as model and reference probabilities.
• principleA principle is a rubric criterion expressing a desirable quality that guides evaluation without being an absolute requirement.
• GSPOGroup sequence policy optimization (GSPO) is a GRPO-derived policy-optimization algorithm that applies importance-sampling corrections at the completion-sequence level.
• length biasLength bias is a systematic preference for outputs of particular lengths that can distort model optimization or evaluation.
• promptA prompt is an input text or instruction provided to a language model to elicit a response.
• process reward modelA process reward model assigns rewards to intermediate steps or partial trajectories in a task or reasoning process.
• verifiable taskA verifiable domain is a problem area in which candidate outputs can be objectively checked for correctness.
• over-refusalOver-refusal is a model behavior in which the model declines requests excessively, including requests it could appropriately answer.
• elicitation theory of post-trainingThe elicitation theory of post-training is the view that post-training primarily extracts and amplifies valuable capabilities already present in a pretrained base model.
• Monte Carlo simulationMonte Carlo estimation approximates an expectation or other quantity by averaging values from samples drawn from the relevant distribution.
• generalized advantage estimationGeneralized Advantage Estimation is a method for estimating advantages by combining multi-step temporal-difference information with a variance-controlling decay parameter.
• returnCumulative reward is the aggregate of rewards collected across an episode or other interaction sequence.
• loss aggregationLoss aggregation is the method of combining per-token or per-sequence losses into a scalar optimization loss.
• rubricA rubric is a structured set of weighted evaluation criteria for judging the quality of a response to a specific task or prompt.
• superficial alignment hypothesisThe superficial alignment hypothesis is the claim that pretraining supplies most knowledge and capability while alignment mainly teaches a model a narrow set of interaction formats or styles.
• loss normalizationLoss normalization is the rescaling of optimization losses or advantage estimates using statistics computed over a specified set of examples.
• online filteringOnline difficulty filtering is the selection of training problems during optimization according to their current difficulty or learning signal.
• rejection samplingRejection sampling is a sampling method that accepts or filters candidate outputs according to a specified criterion or scoring procedure.
• trajectory distributionA trajectory distribution is the probability distribution over complete interaction sequences induced by an initial-state distribution, policy, and environment dynamics.
• preference aggregationPreference aggregation is the process of combining preference judgments from multiple people, contexts, or situations into a collective representation.
• Von Neumann-Morgenstern utility theoremThe Von Neumann–Morgenstern utility theorem states conditions under which rational preferences over lotteries can be represented by an expected-utility function.
• data collectionData acquisition is the process of obtaining data from external providers or internal collection operations for a defined purpose.
• data providerA data vendor is an external provider that organizes people or processes to produce labeled or curated data for a customer.
• overfittingOverfitting is a training condition in which a model fits its training data or objective too closely and generalizes poorly.
• output layerA scalar head is a model output component that maps hidden representations or logits to a single numerical prediction.
• sycophancySycophancy is a model behavior that favors agreement or flattering responses over truthful or independently justified responses.
• alignmentAlignment is the process of shaping a model's behavior and outputs to conform to desired human or task-related objectives.
• RLAIFReinforcement learning from AI feedback is a reinforcement-learning alignment technique that uses AI-generated evaluative feedback as its reward or preference signal.
• data driftDistribution mismatch is a discrepancy between the probability distribution that generated training data and the distribution of the policy being optimized.
• Bellman equationA Bellman equation recursively relates a value or action-value function to immediate reward and the value of successor states.
• action-value functionA Q-function maps a state-action pair to a scalar estimate of expected future reward.
• proxy reward functionA proxy reward is a reward signal that approximates a desired human or task objective rather than measuring it directly.
• instruct modelAn instruction-tuned model is a language model adapted through training on instructions paired with desired responses.
• constitution-as-code documentA constitution is a human-written set of principles specifying aspects of model behavior to assess during critique.
• implicit rewardAn implicit reward is a reward value inferred from a preference-optimization objective rather than predicted by a separately trained reward model.
• on-policy distillationOn-policy distillation is distillation in which training examples or trajectories are sampled from the current student policy and evaluated against a teacher policy.
• off-policy distillationOffline knowledge distillation trains a student on data or outputs generated in advance by a teacher rather than by the current student policy.
• chain-of-thought promptingA reasoning trace is a sequence of generated textual intermediate steps used while solving a problem before reaching an answer.
• CISPOClipped importance sampling policy optimization (CISPO) is a reinforcement-learning algorithm that clips importance weights to preserve token-level gradient signals.
• Constitutional AIConstitutional AI is a model-training approach that uses a set of principles to guide and evaluate model behavior.
• length normalizationLength normalization rescales an objective, score, or gradient according to the number of tokens or steps in an output.
• per-sequence normalizationPer-sequence normalization averages each sequence's token losses so that every sequence contributes equally regardless of length.
• per-token normalizationPer-token normalization averages losses across all active tokens so that every token contributes equally.
• special tokenA special token is a reserved token that marks structural or control information in a model's input or output sequence.
• gradientA gradient is a vector of derivatives indicating how a scalar function changes with respect to its parameters.
• benchmark contaminationDataset contamination is the inclusion of evaluation examples or closely related material in a model's training data, invalidating independent evaluation.
• completionA candidate completion is a generated language-model output considered for acceptance, ranking, or further training.
• persona vectorA persona vector is a vector representation used to encode or manipulate a model's behavioral persona or character traits.
• OpenAI Model SpecA model specification is a written behavioral specification defining intended behaviors and principles for a language model.
• inverse reinforcement learningInverse reinforcement learning is a technique that infers an agent's reward function from observed behavior or trajectories.
• assistant axisAn assistant axis is a learned behavioral direction associated with the characteristics of an AI assistant.
• prompt maskingPrompt masking is the exclusion of prompt-token positions from the training loss while retaining response-token positions.
• constrained decodingConstrained decoding is a generation technique that restricts the tokens or structures a model may produce to satisfy specified validity constraints.
• three-stage rlhf recipeA three-stage RLHF recipe is a workflow consisting of instruction tuning, reward-model training, and reinforcement-learning optimization.
• preference marginA preference margin is the numerical difference in reward or rating values between two compared outputs, representing the strength of preference.
• response-formatting rewardA format reward is a reward signal assigned according to whether a model output follows a required structural presentation convention.
• sequence-level importance ratioA sequence-level importance ratio is an importance-sampling weight computed for an entire generated sequence rather than independently for its tokens.
• exact-match accuracyExact match is an evaluation metric that counts an output as correct only when it exactly matches the required answer string or format.
• labeling guidelineAnnotation guidelines are documented instructions that specify how annotators should perform a data-labeling task, including priorities and decision criteria.
• mode coveringMode covering is a distribution-approximation tendency that spreads probability across the major modes of a target distribution.
• mode seekingMode seeking is a distribution-approximation tendency that concentrates probability on one or a few high-probability modes of a target distribution.
• temporal-difference errorA temporal-difference residual is the difference between an observed immediate reward plus a bootstrapped next-state value and the current value prediction.
• impossibility theoremAn impossibility theorem proves that no rule can simultaneously satisfy a specified set of desirable conditions.
• next-token predictionNext-token prediction is a language-modeling objective that trains a model to predict the token following a given context.
• pretrainingPretraining is the initial large-scale optimization of a model on broad data, commonly through next-token prediction.
• generative reward modelA generative reward model is a reward model that generates evaluative judgments or preference assessments rather than only emitting a scalar score.
• GoodhartingGoodhart's law is the principle that a measure used as an optimization target can cease to be a reliable measure of the underlying goal.
• alignment trainingValue alignment is the conformity of an intelligent system's objectives and behavior to the values held by the people or population it is intended to serve.
• format-agnostic evaluationFormat-agnostic evaluation assesses a model's answer without requiring a rigid output syntax unrelated to the answer's substance.
• code executionCode execution is the process of running source code to obtain its computational behavior or result.
• trust regionA trust region is a constrained neighborhood of parameter values within which an optimization update is permitted to make reliable progress.
• preference displacementPreference displacement is a shift in probability mass away from responses represented in preference-training data toward unaddressed model behaviors.
• implicit feedbackImplicit feedback is information about user preferences inferred from users' actions rather than explicit evaluations.
• monte carlo reward estimateA Monte Carlo return estimate computes a return from rewards observed along complete sampled trajectories rather than bootstrapping from a value estimate.
• online DPOOnline DPO is a DPO variant that samples new completions from the current model during training and incorporates preference signals at training time.
• generation-based evaluationGeneration-based evaluation assesses a model by scoring complete responses it generates rather than selecting from a fixed answer representation.
• model collapseModel collapse is progressive degradation of a model's effective training distribution and generalization caused by repeated training on model-generated data.
• sigmoidThe sigmoid function maps a real-valued input to a number between zero and one using the function 1/(1+e^-x).
• zero-shot learningZero-shot evaluation measures a model on questions or tasks without providing task-specific examples in the prompt.
• group-relative advantage normalizationGroup-relative advantage normalization computes normalized advantage estimates relative to other sampled outputs in the same group.
• multi-armed banditA bandit problem is a sequential decision problem in which an action produces an outcome without a multi-step state-transition sequence.
• length penaltyA length penalty is an objective term that reduces the optimization preference for outputs with undesired lengths.
• prefix biasPrefix bias is a bias in which the beginning of a model completion disproportionately influences a preference judgment.
• surrogate objectiveA surrogate objective is an alternative optimization objective designed to approximate or constrain improvement of a target objective.
• progressive length schedulingProgressive length scheduling is a training procedure that changes the targeted or permitted output length over successive optimization stages.
• log probabilityA log probability is the logarithm of the probability assigned to an event or token by a probabilistic model.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.