Chapter 10: Fine-tuning large language models with PPO
The introduction explains why fluent medical chatbot responses may be unsafe and how reinforcement learning can use human judgments to improve alignment.
After this chapter you will be able to
- Fluency and safety gap. Explain why a fluent, medically informed chatbot can still respond irresponsibly, using the patient-support examples.
- Human judgments for alignment. Explain how reinforcement learning can help address the alignment problem using human judgments about responses, and distinguish that feedback from next-token training.
- Language as reinforcement learning. Describe how language generation fits the reinforcement learning framework in this section.
- Next-token prediction mismatch. Explain why next-token prediction alone does not explicitly optimize the qualities desired in an assistant's responses.
- Human preference between responses. Explain how the mental health example illustrates a preference that reinforcement learning from human feedback can express.
- Text Inputs and Tokens. Describe what a language model learns during pretraining and how input text becomes token IDs.
- Embeddings and Position. Explain how token embeddings and position information form the transformer’s input representations.
- Transformer Contextual Refinement. Explain how transformer layers use attention and feed-forward networks to refine token representations in context.
- Next-Token Decoding Process. Trace how scores become a selected token and how repeated selection produces a response.
- Language Model Policy. Explain why the language model can be viewed as a stochastic policy over tokens.
- State Action and Transition. Trace how a language-generation state changes when the model chooses a token.
- Language Model as Policy. Explain how the language model acts as a stochastic policy when generating a token.
- Human Preference Reward. Explain why RLHF learns a reward function from human preferences for language responses.
- Episode and Sparse Reward. Describe an RLHF episode and explain why its reward timing makes learning harder.
- Three-stage RLHF pipeline. Describe the three stages of RLHF and how they relate to one another.
- Imitation from Expert Responses. Describe SFT as imitation learning from curated query-response pairs, using the medical chatbot to illustrate the behavior it teaches.
- Response-Only Label Masking. Explain how an SFT example is formatted and why loss is computed on response tokens rather than prompt or padding tokens.
- Token Prediction and Stopping. Trace how the model learns an ideal answer token by token and explain why the end-of-sequence token is a target.
- Limits of Demonstration Learning. Explain what learning only from ideal answers leaves out, including response preferences and recovery from mistakes.
- Preference Based Reward Learning. Explain why human comparisons are used to learn a reward function for medical assistant responses rather than relying only on imitation or a hand-written formula.
- Context Dependent Medical Preferences. Explain why the chest-pain example favors a safety-first response and how the intended role affects preference labels.
- Reward Model Scoring Architecture. Describe how the reward model processes a prompt and complete response to produce a preference score.
- Pairwise Ranking And Margin. Trace how chosen and rejected response scores drive reward-model training, including the role of the margin.
- Preference Data Quality Limits. Explain how labeling quality and example coverage limit the learned reward function.
- PPO alignment training loop. Trace how PPO uses generated responses and a learned reward model to improve an SFT policy, and identify the language-generation actions, state, and episode.
- Clipped policy update. Explain how PPO's probability ratio and clipped objective encourage better actions while limiting incentives for large updates.
- Reward shaping and drift. Distinguish the reward model score, optional shaping rewards, and KL penalty, and explain how KL protection differs from PPO clipping.
- Actor critic advantage. Explain how the value head estimates expected reward and how that estimate determines whether generated tokens are encouraged or discouraged.
- Patient support safety constraints. Describe the chatbot’s safety constraints for diagnosis, escalation, and patient communication.
- Stagewise Learning Rate Pattern. Explain how learning rates change across SFT, reward model training, and PPO, and relate the stages to their roles.
- Safety check categories. Describe the four kinds of language or patterns checked by the case study’s safety configuration, with examples.
- Safety signal purpose and limits. Explain how the safety rules are used and why they do not replace a trained reward model or clinical review.
- Supervised fine-tuning purpose. Explain how patient questions paired with ideal responses prepare the model before reinforcement learning, and distinguish this preparation from later preference and safety optimization.
- Prompt and response masking. Explain why the SFT dataset masks prompt labels while retaining the prompt as context for predicting the assistant response.
- Reward model architecture. Explain what the reward model scores and why its reward head uses the last token’s hidden state.
- Pairwise ranking loss. Explain how the pairwise ranking loss encourages the model to distinguish preferred responses from rejected ones.
- Separate reward model copy. Describe how the reward model is initialized and why it is kept separate from the policy model.
- PPO Training Components. Describe the roles of the policy, value head, reference model, and reward model, and how the trainer coordinates them.
- Single-Turn Advantage Estimation. Explain why the return equals the reward in this chatbot and how its advantage is calculated.
- Sampling for Rollout Exploration. Explain how top-p sampling and temperature produce varied but coherent responses during PPO experience collection.
- Independent Policy and Reference. Explain why PPO uses separate SFT-model copies for the trainable policy and frozen reference.
- SFT and PPO gains. Explain how SFT and PPO contributed differently to the medical chatbot’s evaluation results.
- Adaptive KL constraint. Explain what the changing KL coefficient indicates and why the constraint matters in this run.
- PPO beyond language models. Explain why PPO applies beyond language models, using the ambulance repositioning example.
What it builds on
- Markov decision process (chapter 2) · The basic framing of decisions, actions, environments, and rewards.
- The Transformer architecture (chapter 9) · The earlier account of transformer architecture provides context for its input representations.
- Attention mechanism: learning what to look at (chapter 9) · Attention provides the basis for mixing information from relevant tokens.
- Why optimizing the policy directly? (chapter 9) · Direct policy optimization provides context for treating a model as a policy.
- Design rewards to be stepwise, whenever possible (chapter 2) · The value of stepwise rewards when designing feedback.
- The REINFORCE algorithm (chapter 9) · The REINFORCE intuition of changing action probabilities based on outcomes.
- Variance reduction through baselines (chapter 9) · The idea of comparing outcomes with a baseline.
- Avoid mismatched scales with reward normalization / balancing (chapter 2) · Balancing reward scales so one signal does not dominate another.
Section by section
Introduction
The introduction explains why fluent medical chatbot responses may be unsafe and how reinforcement learning can use human judgments to improve alignment.
- Fluency and safety gap: Explain why a fluent, medically informed chatbot can still respond irresponsibly, using the patient-support examples.
Why it matters: A convincing answer can still fail a patient who needs urgent or empathetic support. - Human judgments for alignment: Explain how reinforcement learning can help address the alignment problem using human judgments about responses, and distinguish that feedback from next-token training.
Why it matters: Human comparisons provide a way to favor more appropriate responses beyond simply producing fluent text.
Key terms: fluency, safety, alignment problem, reinforcement learning, human feedback, next-token training
10.1 A New Frontier for Reinforcement Learning: Language
This section introduces language generation as a reinforcement learning problem and explains why human preferences matter beyond next-token prediction.
- Language as reinforcement learning: Describe how language generation fits the reinforcement learning framework in this section.
Why it matters: This mapping connects familiar reinforcement learning ideas to language generation. - Next-token prediction mismatch: Explain why next-token prediction alone does not explicitly optimize the qualities desired in an assistant's responses.
Why it matters: Fluency alone does not guarantee that a response meets human needs. - Human preference between responses: Explain how the mental health example illustrates a preference that reinforcement learning from human feedback can express.
Why it matters: Human preferences can distinguish a better response even when more than one answer is plausible.
Key terms: tokens, actions, environment, reward, human judgment, next-token prediction, statistical patterns, safety, helpfulness, appropriateness
10.2 Overview of Large Language Models Architecture
This section traces how a language model turns text into contextual representations and generates tokens, connecting that process to a stochastic RL policy.
- Text Inputs and Tokens: Describe what a language model learns during pretraining and how input text becomes token IDs.
Why it matters: This explains both the model’s basic training task and the units it processes. - Embeddings and Position: Explain how token embeddings and position information form the transformer’s input representations.
Why it matters: The model needs numerical representations that capture both the tokens and their sequence. - Transformer Contextual Refinement: Explain how transformer layers use attention and feed-forward networks to refine token representations in context.
Why it matters: Contextual refinement lets the same token be represented differently in different messages. - Next-Token Decoding Process: Trace how scores become a selected token and how repeated selection produces a response.
Why it matters: This shows how the architecture produces a response one token at a time. - Language Model Policy: Explain why the language model can be viewed as a stochastic policy over tokens.
Why it matters: This connects language generation to the policy-based RL methods used in the chapter. - Preserving Language Abilities: Explain why alignment should not come at the expense of the model’s original language abilities.
Why it matters: Safer behavior is useful only if the model remains capable of useful conversation.
Key terms: pretraining, tokenization, tokens, vocabulary, embeddings, dense vector, position, transformer, attention, feed-forward neural network
10.3 Mapping Language Model to a Markov Decision Process
This section maps language generation to a Markov Decision Process and explains the challenges of rewarding complete responses.
- State Action and Transition: Trace how a language-generation state changes when the model chooses a token.
Why it matters: This mapping makes token-by-token generation a sequential decision process. - Language Model as Policy: Explain how the language model acts as a stochastic policy when generating a token.
Why it matters: The policy is the part of the language-generation process that makes the token choice. - Human Preference Reward: Explain why RLHF learns a reward function from human preferences for language responses.
Why it matters: Human preferences provide a way to reward response qualities that are difficult to specify in a formula. - Episode and Sparse Reward: Describe an RLHF episode and explain why its reward timing makes learning harder.
Why it matters: A single final score gives less immediate feedback about the individual choices that produced a response.
Key terms: state, action, token, vocabulary, transition, policy, stochastic policy, categorical distribution, sampling, reward function
10.4 The RLHF Pipeline: Three Stages to Alignment
This section introduces RLHF as a sequential, three-stage process for aligning language models.
- Three-stage RLHF pipeline: Describe the three stages of RLHF and how they relate to one another.
Why it matters: The sequence provides a framework for understanding how the later stages of alignment fit together.
Key terms: RLHF, supervised learning, reward modeling, reinforcement learning
10.5 Supervised Fine-Tuning (SFT)
This section explains how supervised fine-tuning teaches a language model to imitate domain-specific responses, how response-only token loss works, and what imitation alone cannot teach.
- Domain-Specific Starting Model: Explain why the medical chatbot starts with an already instruction-tuned model and fine-tunes it further.
Why it matters: The starting model provides useful existing abilities while leaving room to adapt to medical-safety requirements. - Imitation from Expert Responses: Describe SFT as imitation learning from curated query-response pairs, using the medical chatbot to illustrate the behavior it teaches.
Why it matters: This explains how examples give the chatbot a foundation for appropriate medical responses. - Response-Only Label Masking: Explain how an SFT example is formatted and why loss is computed on response tokens rather than prompt or padding tokens.
Why it matters: Label masking focuses learning on the assistant’s answer while preserving the prompt as context. - Token Prediction and Stopping: Trace how the model learns an ideal answer token by token and explain why the end-of-sequence token is a target.
Why it matters: Token-level targets teach both what to say and when the answer is complete. - Limits of Demonstration Learning: Explain what learning only from ideal answers leaves out, including response preferences and recovery from mistakes.
Why it matters: A strong imitative foundation alone does not teach the chatbot to rank reasonable answers or recover from its own errors.
Key terms: pre-trained large language model, instruction-tuned, domain-specific goals, supervised fine-tuning, imitation learning, behavioral cloning, query-response pairs, system prompt, user query, ideal response
10.6 Learning a Reward Model from Human Preferences
This section explains how a reward model learns to score medical assistant responses from human preference comparisons and why its behavior depends on the preference data.
- Preference Based Reward Learning: Explain why human comparisons are used to learn a reward function for medical assistant responses rather than relying only on imitation or a hand-written formula.
Why it matters: Preference learning provides a way to reward response qualities that are difficult to specify directly. - Context Dependent Medical Preferences: Explain why the chest-pain example favors a safety-first response and how the intended role affects preference labels.
Why it matters: The same response can be preferred or rejected depending on the assistant’s purpose and the raters’ instructions. - Reward Model Scoring Architecture: Describe how the reward model processes a prompt and complete response to produce a preference score.
Why it matters: The architecture turns a complete response into a numerical estimate of human preference. - Pairwise Ranking And Margin: Trace how chosen and rejected response scores drive reward-model training, including the role of the margin.
Why it matters: The training objective teaches relative preference instead of treating arbitrary example scores as fixed targets. - Preference Data Quality Limits: Explain how labeling quality and example coverage limit the learned reward function.
Why it matters: A well-built scoring model can still reward unsafe behavior if its examples and labels are inadequate.
Key terms: SFT, reward model, human preferences, reward function, chosen response, rejected response, human preference labels, safety-first, transformer backbone, reward head
10.7 Proximal Policy Optimization: From REINFORCE to PPO
This section explains how PPO uses learned preference rewards to fine-tune a language model while controlling policy updates and drift from the supervised fine-tuned model.
- PPO alignment training loop: Trace how PPO uses generated responses and a learned reward model to improve an SFT policy, and identify the language-generation actions, state, and episode.
Why it matters: This explains how preference judgments become improvements to the chatbot rather than merely scores. - Clipped policy update: Explain how PPO's probability ratio and clipped objective encourage better actions while limiting incentives for large updates.
Why it matters: Controlled updates reduce the risk of a policy changing abruptly or exploiting flaws in the reward model. - Reward shaping and drift: Distinguish the reward model score, optional shaping rewards, and KL penalty, and explain how KL protection differs from PPO clipping.
Why it matters: The two protections address different risks: a large individual update and cumulative departure from trusted SFT behavior. - Actor critic advantage: Explain how the value head estimates expected reward and how that estimate determines whether generated tokens are encouraged or discouraged.
Why it matters: Advantage relates a response's reward to what the model expected, giving the policy a direction for learning. - Combined PPO loss: Describe the purposes of the policy loss, value loss, and entropy bonus in PPO's combined training loss.
Why it matters: The combined loss trains both response behavior and reward prediction while preserving exploration.
Key terms: policy model, reward model, SFT, token, state, episode, probability ratio, old policy, advantage, clipped objective
10.8 Build a Medical Patient Support Chatbot with RLHF
The section introduces a safety-sensitive medical chatbot case study and explains the safety requirements and practical dataset choices guiding its RLHF pipeline.
- Patient support safety constraints: Describe the chatbot’s safety constraints for diagnosis, escalation, and patient communication.
Why it matters: These boundaries define what safer patient support should look like in the case study. - Safety driven case design: Explain how the safety constraints and the small custom dataset shape the case study.
Why it matters: The case study’s safety requirements affect its design, while its small scale makes the complete pipeline practical to study.
Key terms: safety constraints, professional consultation, emergencies, empathy, reward model, reward shaping, final evaluation, custom dataset
10.8.1 Pipeline configuration
This section highlights the decreasing learning rates across the RLHF pipeline stages and relates them to each stage’s role.
- Stagewise Learning Rate Pattern: Explain how learning rates change across SFT, reward model training, and PPO, and relate the stages to their roles.
Why it matters: This pattern connects the size of training updates to what each stage is trying to accomplish.
Key terms: learning rate, SFT, reward model training, PPO, preference-scoring function, SFT policy
10.8.2 Safety signals
This section describes keyword and pattern checks used as lightweight safety signals in a medical chatbot case study, along with their limitations.
- Safety check categories: Describe the four kinds of language or patterns checked by the case study’s safety configuration, with examples.
Why it matters: These checks make concrete what the case study looks for when assessing chatbot safety. - Safety signal purpose and limits: Explain how the safety rules are used and why they do not replace a trained reward model or clinical review.
Why it matters: The checks are useful for inspecting the case study, but their simplicity limits what they can replace.
Key terms: emergency language, empathy language, professional referral language, dangerous patterns, reward shaping, evaluation, structured signals, trained reward model, clinical review
10.8.3 Stage 1: Supervised fine-tuning
This section explains the medical chatbot’s supervised fine-tuning stage, its datasets, and the training pipeline.
- Supervised fine-tuning purpose: Explain how patient questions paired with ideal responses prepare the model before reinforcement learning, and distinguish this preparation from later preference and safety optimization.
Why it matters: This distinction shows what example-based preparation achieves and what remains for later stages. - Prompt and response masking: Explain why the SFT dataset masks prompt labels while retaining the prompt as context for predicting the assistant response.
Why it matters: Masking directs supervised training toward the answer the assistant is supposed to produce. - Preference dataset construction: Describe how the preference dataset pairs and prepares chosen and rejected responses for the reward model.
Why it matters: The paired examples provide the reward model with contrasting responses to the same patient query. - SFT training pipeline: Describe the main steps and training safeguards in the SFT pipeline, including its optimizer, learning-rate schedule, and handling of NaN losses.
Why it matters: These steps prepare the data and model for training while preventing invalid losses from corrupting updates.
Key terms: supervised fine-tuning, ideal assistant responses, safety style, label masking, prompt, assistant response, preference dataset, chosen response, rejected response, reward model
10.8.4 Stage 2: reward model training
This section explains how the medical chatbot’s reward model and supplementary shaping signals are constructed, trained, and validated.
- Supplementary reward shaping: Explain how the shaping rewards supplement the learned reward and why their scale is deliberately small.
Why it matters: Reward shaping adds guidance while keeping the learned preferences central. - Reward model architecture: Explain what the reward model scores and why its reward head uses the last token’s hidden state.
Why it matters: This explains how a language model turns a complete interaction into a reward signal. - Pairwise ranking loss: Explain how the pairwise ranking loss encourages the model to distinguish preferred responses from rejected ones.
Why it matters: The ranking objective turns response preferences into a training signal for the reward model. - Preference ranking validation: Describe how reward model validation accuracy is measured and what the reported result suggests.
Why it matters: This check shows whether the reward model orders the validation preferences correctly. - Separate reward model copy: Describe how the reward model is initialized and why it is kept separate from the policy model.
Why it matters: Keeping the instances separate protects the policy’s starting point during reward model training.
Key terms: reward shaping, shaping rewards, learned reward, reward model, query, response, last token's hidden state, causal attention, reward head, pairwise ranking loss
10.8.5 Stage 3: PPO fine-tuning
This section explains how the medical chatbot’s PPO stage collects experience, computes advantages, updates its policy, and evaluates safety.
- PPO Training Components: Describe the roles of the policy, value head, reference model, and reward model, and how the trainer coordinates them.
Why it matters: These components turn generated chatbot responses into experience the policy can learn from. - Single-Turn Advantage Estimation: Explain why the return equals the reward in this chatbot and how its advantage is calculated.
Why it matters: This simplification makes the case study’s PPO advantage calculation easy to inspect. - Sampling for Rollout Exploration: Explain how top-p sampling and temperature produce varied but coherent responses during PPO experience collection.
Why it matters: Varied rollouts give PPO responses to compare without relying on implausible tokens. - Generated Token Log Probabilities: Describe what the log probability computation measures for generated tokens.
Why it matters: The update needs a measure of how likely the policy was to produce the collected tokens. - Independent Policy and Reference: Explain why PPO uses separate SFT-model copies for the trainable policy and frozen reference.
Why it matters: A genuinely separate reference is needed for the KL-based safety constraint to have meaning. - Comparative Safety Evaluation: Describe how the case study uses safety signals to compare the base, SFT, and PPO models, and state the evaluation’s limitation.
Why it matters: A consistent comparison helps assess whether PPO made the chatbot safer without treating automated scoring as clinical review.
Key terms: policy model, value head, reference model, reward model, PPO trainer, single-turn, return, reward, advantage, value
10.8.6 Evaluation
Evaluates the medical chatbot’s RLHF results and illustrates how PPO supports stable policy improvement beyond language models.
- SFT and PPO gains: Explain how SFT and PPO contributed differently to the medical chatbot’s evaluation results.
Why it matters: The stages contribute differently: examples produce a large initial gain, while PPO improves critical safety behaviors further. - Compute and alignment results: Describe how the two compute configurations compared and what the comparison suggests.
Why it matters: The comparison shows a potential benefit of additional training resources without proving a universal scaling law. - Dangerous content preservation: State what happened to the dangerous content rate across the evaluated models and configurations.
Why it matters: The evaluation checks whether improving other behaviors preserves an existing safety property. - Adaptive KL constraint: Explain what the changing KL coefficient indicates and why the constraint matters in this run.
Why it matters: A reward-optimized chatbot can exploit its reward model unless its behavior is kept from drifting too far. - PPO beyond language models: Explain why PPO applies beyond language models, using the ambulance repositioning example.
Why it matters: PPO’s stable policy improvement is useful for operational decisions as well as chatbot responses.
Key terms: SFT, PPO, emergency recognition, urgency match, low-compute run, high-compute run, scaling law, alignment, dangerous content rate, base model
10.9 Summary
Where this is used later
- Teaching language models to reason on financial reports (chapter 11) · Language generation can be mapped to an MDP in which tokens are actions.
- Robotics and physical control (chapter 11) · The mapping of language modeling to states and actions.
- (chapter intro) (chapter 11) · The learned reward model based on human preferences that a verifier can replace.
- LoRA: fine-tuning large models with tiny adapters (chapter 11) · The PPO context in which a reference policy and KL penalty help control policy drift.
- GRPO: group relative policy optimization (chapter 11) · PPO’s clipped policy updates and value-based advantage baseline
- Five lessons from training GRPO at scale (chapter 11) · Considers lessons from training at scale.
- Diagnosing training stability with KL divergence (chapter 11) · Uses KL divergence to diagnose training stability.
- Operations research and combinatorial optimization (chapter 11) · Considers policy-based methods in operations research and combinatorial optimization.
Concepts explained in this chapter
- RLHFReinforcement learning from human feedback uses human feedback, often through a learned preference model, to guide a model's behavior.
- reward modelA reward model predicts a numerical score for an output to represent its desirability or alignment with preferences.
- KL regularizationA regularization term based on Kullback–Leibler divergence that discourages a policy from moving too far from a reference policy.
- supervised fine-tuningSupervised fine-tuning is the training of a pre-trained model on labeled examples so that it produces desired outputs for given inputs.
- clipped surrogate objectiveA PPO objective that clips the policy probability ratio to limit the effect of policy updates.
- large language modelA machine-learning model trained on large text corpora to model and generate language.
- PPO clippingPPO clipping restricts the optimization benefit of policy probability ratios that exceed a chosen interval.
- importance weightA probability ratio in PPO is the updated policy's probability for an action divided by the old policy's probability for that action.
- policy modelA policy model is the trainable model that generates actions according to a policy.
- reference modelA reference model is a fixed model whose policy distribution provides a comparison point during policy training.
- tokenA token is a discrete unit of text, such as a word, word fragment, punctuation mark, or symbol, processed by a language model.
- policyA policy is a decision rule, possibly probabilistic, that assigns actions or action probabilities to states for an agent.
- vocabularyA language model’s vocabulary is the set of tokens it can represent and select as output.
- next-word predictionA language-model training approach that predicts the next token from the preceding sequence.
- reward model trainingReward modeling trains a model to estimate the desirability of responses from preference comparisons.
- Proximal Policy OptimizationPPO is a policy-gradient reinforcement-learning algorithm that constrains policy updates to maintain stable improvements.
- imitation learningImitation learning trains an agent to reproduce behavior demonstrated by an expert or another policy.
- action spaceAn action space is the set or structured specification of actions that an agent can submit to an environment.
- decoding strategyDecoding is the procedure that converts a model’s output scores into a selected token or sequence.
- embeddingAn embedding is a dense vector representation of an item, such as a token, learned by a model.
- logitA logit is an unnormalized real-valued score that softmax transforms into an action probability.
- probability distributionA probability distribution specifies the probabilities assigned to the possible values of a random variable.
- stochastic policyA stochastic policy is a decision rule that assigns a probability distribution over possible actions for each state.
- tokenizationTokenization is the process of dividing text into discrete units called tokens.
- transformer architectureA neural-network architecture built around self-attention, position-wise feedforward transformations, residual connections, and normalization.
- nucleus samplingNucleus sampling draws a token from the smallest set of highest-probability tokens whose cumulative probability reaches a chosen threshold.
- value headA language-model value head estimates the expected reward for a response generated by a language model.
- actionAn action is a choice made by an agent that can affect the subsequent state of its environment.
- episodeA finite sequence of agent-environment interactions beginning from an initial state and ending when a task terminates.
- reward functionA reward function is a mapping that assigns numerical feedback to states, actions, transitions, or outcomes in a decision process.
- stateA state describes the relevant condition of an environment at a particular point in a sequential decision process.
- reinforcement learningReinforcement learning is a machine-learning paradigm in which an agent learns to choose actions through interactions with an environment and feedback about their consequences.
- learning rateA learning rate is a numerical step-size parameter that controls how strongly an update changes an estimate or model parameter.
- entropy regularizationEntropy regularization adds an entropy-based term to the objective to encourage diversity in a policy's action distribution.
- supervised learningA machine-learning technique that learns mappings from inputs to target outputs using labeled examples.
- feedforward networkA feedforward neural network is a neural network in which information flows from input to output without recurrent connections.
- log probabilityThe logarithm of the probability or probability density assigned to an outcome by a probability model.
- discrete action spaceA discrete action space is a finite set of distinct actions from which an agent selects.
- sparse rewardA sparse reward is a feedback signal provided at only a small number of points, rather than continuously throughout a task.
- transition probabilityA transition probability is the probability of moving from one state to another in a stochastic system.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
