What you will learn in Applied Reinforcement Learning
Business optimization and LLM fine-tuning
Chapter by chapter
Chapter 1: Real-world Decision Making with Reinforcement Learning
The introduction frames business decisions as repeated choices under uncertainty and connects that challenge to reinforcement learning.
What you learn in chapter 1 →Chapter 2: Markov Decision Process: Turning Problems intro Solvable Models
This introduction frames Markov decision processes as a way to break down and reconnect complex business decisions over time.
What you learn in chapter 2 →Chapter 3: Design Custom Environments for Reinforcement Learning Algorithms
This introduction explains why reinforcement learning agents need simulated environments for safe trial-and-error learning in business settings.
What you learn in chapter 3 →Chapter 4: Perfect Knowledge, Optimal Policy: Dynamic Programming
This introduction contrasts model-based and model-free methods and situates dynamic programming as a recursive approach to solving Markov decision processes with a known model.
What you learn in chapter 4 →Chapter 5: Contextual Bandit: Optimizing Stochastic One-step Decisions
This introduction motivates bandits as experience-based decision methods and distinguishes pure from contextual bandits.
What you learn in chapter 5 →Chapter 6: Tabular Reinforcement Learning
The introduction positions temporal-difference learning as a foundation of reinforcement learning and briefly contrasts tabular and approximation-based methods.
What you learn in chapter 6 →Chapter 7: Monte Carlo Tree Search: Searching with Reinforcement Learning Principles
This introduction presents Monte Carlo tree search as a blend of simulation, planning, and learning for exploring sequential decisions under uncertainty.
What you learn in chapter 7 →Chapter 8: Deep Q-Networks for High-dimensional Data
This introduction uses radiotherapy planning to motivate moving beyond tabular reinforcement learning for complex, high-dimensional decisions.
What you learn in chapter 8 →Chapter 9: The Calculus of Decisions: Policy Gradient Methods
This introduction motivates direct policy optimization using the challenges of continuous, coupled data center cooling control.
What you learn in chapter 9 →Chapter 10: Fine-tuning large language models with PPO
The introduction explains why fluent medical chatbot responses may be unsafe and how reinforcement learning can use human judgments to improve alignment.
What you learn in chapter 10 →Chapter 11: Teach LLMs to reason with Verifiable Feedback using GRPO
This introduction explains when a programmatic verifier can replace a learned reward model for language-model tasks.
What you learn in chapter 11 →Chapter 12: Building Reinforcement Learning systems: LLM-guided RL agent for portfolio management
This introduction frames practical RL as a systems-engineering challenge and illustrates it with an LLM-guided portfolio manager.
What you learn in chapter 12 →Concepts explained in this book
- reward functionA reward function is a mapping that assigns numerical feedback to states, actions, transitions, or outcomes in a decision process.
- policyA policy is a decision rule, possibly probabilistic, that assigns actions or action probabilities to states for an agent.
- value functionA value function estimates the expected cumulative return from a state or state-action pair under a specified policy or optimal behavior.
- action-value functionAn action-value function maps a state-action pair to the expected return from taking the action and then following a policy.
- action spaceAn action space is the set or structured specification of actions that an agent can submit to an environment.
- stateA state describes the relevant condition of an environment at a particular point in a sequential decision process.
- Markov decision processA Markov decision process is a formal model of sequential decision-making in which an agent selects actions in states, receives rewards, and causes probabilistic transitions whose relevant future behavior depends on the current state.
- environmentAn environment is the external system or setting with which an agent interacts and whose responses produce consequences for the agent's actions.
- REINFORCEA Monte Carlo policy-gradient algorithm that updates policy parameters using sampled returns and the gradient of the log probability of selected actions.
- DQNDeep Q-network (DQN) is a reinforcement-learning algorithm that uses a neural network to approximate action values for Q-learning.
- transition probabilityA transition probability is the probability of moving from one state to another in a stochastic system.
- reward signalA reward is a numerical feedback signal indicating the desirability of an outcome or transition after an agent's action.
- advantage functionAdvantage measures how much an action's expected return exceeds or falls below the value of its state under a policy.
- Bellman equationA Bellman equation expresses the value of a state or state-action pair as an immediate reward plus the appropriately discounted value of future states.
- policy gradient methodA reinforcement-learning method that directly optimizes the parameters of a policy using the gradient of expected return.
- upper confidence boundAn upper confidence bound method selects actions using an optimistic estimate that combines estimated reward with an uncertainty-based confidence bonus.
- replay bufferA replay buffer is a storage structure that retains interaction records for later processing or learning updates.
- experience replayExperience replay is a reinforcement-learning technique that stores interaction transitions and trains on randomly sampled mini-batches to reduce temporal correlation.
- TD-learningTemporal-difference learning updates value estimates incrementally using observed rewards and estimates of subsequent values.
- Markov propertyThe Markov property is the condition that the distribution of future states and rewards depends on the current state and action rather than on unrepresented earlier history.
- objective functionA mathematical quantity that a learning or optimization procedure seeks to minimize or maximize.
- AI agentAn entity in reinforcement learning that observes an environment, selects actions, and improves its behavior from feedback.
- transformer architectureA neural-network architecture built around self-attention, position-wise feedforward transformations, residual connections, and normalization.
- SARSA(λ)SARSA(λ) is an on-policy temporal-difference algorithm that combines SARSA updates with eligibility traces.
- state spaceA state space is the set of possible states of an environment in a decision process.
- actionAn action is a choice made by an agent that can affect the subsequent state of its environment.
- RLHFReinforcement learning from human feedback uses human feedback, often through a learned preference model, to guide a model's behavior.
- curriculum learningCurriculum learning trains a model by beginning with simpler tasks or environments and progressively increasing their difficulty.
- feedforward networkA feedforward neural network is a neural network in which information flows from input to output without recurrent connections.
- epsilon-greedyAn epsilon-greedy strategy selects a random action with probability epsilon and otherwise selects an action with the highest estimated value.
- tabular methodTabular reinforcement learning stores value estimates or policy information explicitly in lookup tables indexed by states or state-action pairs.
- radiation doseThe amount of ionizing radiation delivered to a patient or specified tissue during treatment.
- radiotherapy treatment planningRadiotherapy treatment planning is the process of selecting radiation delivery parameters over treatment sessions to control a tumor while limiting exposure to healthy tissue.
- organ at riskAn organ at risk is a healthy organ near a radiation target whose exposure must be kept below a clinical dose limit.
- radiation toxicityHarmful tissue or organ injury caused by radiation exposure.
- radiotherapyRadiotherapy is a cancer treatment that uses ionizing radiation to damage and destroy tumor cells.
- tumorA tumor is an abnormal mass of tissue formed by uncontrolled cell growth and targeted for destruction in radiotherapy.
- greedy policyA greedy policy selects the product with the lower current stock level, resolving equal stock levels by choosing scooter.
- generalized policy iterationGeneralized policy iteration is a reinforcement-learning framework that alternates policy evaluation with policy improvement until the policy stabilizes or reaches an optimum.
- eligibility traceAn eligibility trace is a decaying record of recently visited state-action pairs that determines how temporal-difference errors are assigned backward through them.
- policy evaluationPolicy evaluation is the process of estimating the value of states or state-action pairs under a fixed policy.
- policy improvementPolicy improvement is the process of deriving a policy that selects actions with higher estimated value than the current policy.
- reward modelA reward model predicts a numerical score for an output to represent its desirability or alignment with preferences.
- temporal-difference errorA temporal-difference error is the difference between a target formed from an immediate reward and a discounted estimate of future value and the current value estimate.
- value iterationValue iteration is a dynamic-programming algorithm that repeatedly applies a Bellman optimality update to approximate optimal state values and derive an optimal policy.
- normal distributionA Gaussian distribution is a continuous probability distribution characterized by a mean and standard deviation.
- Monte Carlo Tree SearchMonte Carlo tree search is a planning algorithm that incrementally builds a search tree by selecting nodes, expanding actions, simulating outcomes, and backpropagating outcome statistics.
- reinforcement learningReinforcement learning is a machine-learning paradigm in which an agent learns to choose actions through interactions with an environment and feedback about their consequences.
- transition dynamicThe rules governing how an environment changes from one state to another after an action, including effects of randomness.
- constraintA restriction that limits the values or combinations allowed in an optimization problem, including which actions are valid in a state.
- Thompson samplingA Bayesian bandit action-selection algorithm that samples each action's expected reward from its posterior distribution and selects the action with the best sample.
- contextual banditA contextual bandit is a sequential decision model in which an agent observes static context, selects an action, and receives an immediate reward without action-induced state transitions.
- large language modelA machine-learning model trained on large text corpora to model and generate language.
- exploration-exploitation trade-offThe exploration-exploitation trade-off is the tension between gathering information through unfamiliar actions and selecting actions currently expected to yield high rewards.
- multi-armed banditA multi-armed bandit is a sequential decision model in which an agent repeatedly chooses among actions and observes their rewards without modeled state transitions.
- dynamic pricingDynamic pricing is the adjustment of prices over time in response to changing demand, inventory conditions, expiration, or competitive factors.
- Monte Carlo simulationMonte Carlo estimation approximates a quantity by averaging values obtained from random samples.
- function approximatorFunction approximation estimates a value function or policy with a parameterized mathematical function instead of storing a separate value for every state or state-action pair.
- on-policy methodOn-policy learning is reinforcement learning in which the policy generating experience is the same policy being evaluated or improved.
- TD(λ)TD(lambda) is a family of temporal-difference methods that combines bootstrapped value updates with eligibility traces to assign credit across multiple steps.
- learning rateA learning rate is a numerical step-size parameter that controls how strongly an update changes an estimate or model parameter.
- returnThe discounted sum of future rewards from a particular time step in a trajectory.
- Group Relative Policy OptimizationA reinforcement-learning algorithm that compares rewards among multiple outputs sampled for the same prompt to estimate relative advantages for policy updates.
- cooling zoneA monitored region of a data center with local thermal and workload conditions that influence cooling controls.
- off-policy methodOff-policy learning is reinforcement learning in which a target policy is evaluated or improved using experience generated by a potentially different behavior policy.
- never switch policyA never switch policy continues producing the configured product until a target inventory is reached and then idles without changing setup.
- setup-aware policyA setup-aware policy uses inventory thresholds and the current production setup to avoid unnecessary product switches while responding to critical shortages.
- backorderA backorder is an unfulfilled customer demand recorded for later fulfillment because available inventory was insufficient.
- holding costHolding cost is the expense incurred for keeping unsold inventory over time.
- inventory levelAn inventory level is the quantity of a product currently held and available for sale or use.
- machine setupA machine setup is the configuration of production equipment required to manufacture a particular product.
- market demandCustomer demand is the quantity of products customers request during a specified period, with stochastic demand varying unpredictably over time.
- Poisson distributionThe Poisson distribution gives the probability of a count of independent events occurring during a fixed interval at a roughly constant average rate.
- production planningProduction planning is the process of deciding what products to manufacture, in what quantities, and when, subject to capacity, demand, and cost considerations.
- setup costA setup cost is the expense incurred when configuring or switching production equipment for a product.
- search treeA search tree is a branching data structure whose nodes represent states or decision points and whose edges represent possible actions or outcomes.
- KL regularizationA regularization term based on Kullback–Leibler divergence that discourages a policy from moving too far from a reference policy.
- supervised fine-tuningSupervised fine-tuning is the training of a pre-trained model on labeled examples so that it produces desired outputs for given inputs.
- transformer encoderA neural-network encoder formed from stacked self-attention and feedforward layers with residual connections and normalization.
- reference modelA reference model is a fixed model whose policy distribution provides a comparison point during policy training.
- entropy regularizationEntropy regularization adds an entropy-based term to the objective to encourage diversity in a policy's action distribution.
- log probabilityThe logarithm of the probability or probability density assigned to an outcome by a probability model.
- multi-head attentionMulti-head attention performs several independent attention computations in parallel and combines their outputs to capture different relationship patterns.
- resource allocation problemA resource allocation problem is a decision problem that assigns limited resources among competing activities to optimize an objective subject to constraints.
- business optimization problemBusiness optimization is the application of analytical and mathematical methods to improve decisions about business operations and resource allocation.
- softmaxSoftmax converts a vector of real-valued scores into nonnegative probabilities that sum to one.
- distillationA model-training technique in which a student model learns to reproduce useful outputs or behavior of a teacher model.
- reset functionAn environment method that initializes a new episode and returns its initial observation or state.
- policy gradient theoremA theorem expressing the gradient of expected policy return using policy score functions and action values without requiring differentiation of the environment dynamics.
- logitA logit is an unnormalized real-valued score that softmax transforms into an action probability.
- inventory managementInventory management is the planning and control of replenishment and stock levels to balance availability and inventory-related costs.
- Q(lambda)Q(lambda) is an off-policy action-value learning algorithm that combines Q-learning targets with eligibility traces and resets traces after non-greedy actions.
- stockoutA stockout is a condition in which available inventory is insufficient to satisfy current demand.
- Markov chainA Markov chain is a stochastic model describing how a system evolves among states when the next state depends probabilistically on the current state.
- Q-learningQ-learning is a model-free reinforcement-learning algorithm that estimates the optimal action-value function through temporal-difference updates using the maximum estimated value of the next state.
- online learningOnline learning updates a model incrementally as data or interaction outcomes become available rather than waiting for a complete dataset or episode.
- tree searchTree search is a planning technique that represents possible action sequences and outcomes as branches of a search tree and uses them to guide decisions.
- 2-opt2-opt is a routing improvement heuristic that replaces two route edges by reconnecting the resulting segments in the opposite order.
- capacitated vehicle routing problemThe capacitated vehicle routing problem is an optimization problem that assigns customer visits to vehicle routes while respecting vehicle capacities and minimizing total travel cost.
- capacity constraintA capacity constraint limits the total demand assigned to a vehicle route to the vehicle's carrying capacity.
- local searchLocal search is an optimization technique that iteratively improves a candidate solution by examining nearby solutions.
- reparameterization trickA stochastic-gradient technique that expresses a random sample as a deterministic transformation of parameter-independent noise.
- sigmoidA differentiable function that maps real numbers to values between zero and one.
- softplus functionA smooth function, log(1 + exp(x)), commonly used to provide differentiable positive-valued outputs or penalties.
- greedy agentA greedy agent always selects the action with the highest current estimated payoff and does not deliberately explore alternatives.
- contextContext is observable information about a decision situation that is available before an action is selected.
- convergenceConvergence is the approach of successive numerical estimates or policies toward a stable limiting result.
- clipped surrogate objectiveA PPO objective that clips the policy probability ratio to limit the effect of policy updates.
- LinUCBA contextual bandit algorithm that models expected reward as a linear function of context and uses upper confidence bounds to balance exploration and exploitation.
- dynamic programmingDynamic programming is an algorithmic technique that solves sequential decision or optimization problems by decomposing them into overlapping subproblems and combining their solutions.
- model-free methodModel-free learning is reinforcement learning in which an agent learns action selection directly from experience without an explicit environment transition model.
- reinforcement learning with verifiable rewardsA reinforcement-learning approach that uses objectively checkable criteria to reward model outputs.
- policy modelA policy model is the trainable model that generates actions according to a policy.
- importance weightA probability ratio in PPO is the updated policy's probability for an action divided by the old policy's probability for that action.
- PPO clippingPPO clipping restricts the optimization benefit of policy probability ratios that exceed a chosen interval.
- beta distributionA continuous probability distribution on the interval from zero to one, commonly used to represent uncertainty about a Bernoulli probability.
- uncertainty bonusAn additive score component that favors options whose estimated values are less certain or based on fewer observations.
- log-derivative trickA calculus identity that rewrites derivatives of probability distributions as expectations involving derivatives of their logarithms.
- score functionThe gradient of the log probability of an action, indicating how parameter changes affect that action's likelihood.
- tokenA token is a discrete unit of text, such as a word, word fragment, punctuation mark, or symbol, processed by a language model.
- Watkins's Q(λ)Watkins's Q(λ) is an off-policy Q-learning variant that resets eligibility traces when a non-greedy action is taken.
- deep neural networkA deep neural network is a neural network with multiple learned layers between its input and output.
- sequential decision-makingA decision process in which actions are chosen over multiple steps while relevant states, outcomes, or future conditions are not fully known.
- target networkA target network is a delayed copy of an action-value network whose parameters are updated periodically and used to compute learning targets.
- verifierA verifier is a procedure that checks a model's answer against a specified correctness criterion or reference answer.
- trajectoryA trajectory is a sampled sequence of states, actions, and outcomes generated by an agent interacting with an environment.
- bootstrappingBootstrapping is an estimation technique that updates a prediction using another prediction or value estimate rather than waiting for a complete realized outcome.
- discrete action spaceA discrete action space is a finite set of distinct actions from which an agent selects.
- episodeA finite sequence of agent-environment interactions beginning from an initial state and ending when a task terminates.
- look-ahead biasBias caused by using information that would not have been available at the time of a decision or prediction.
- discount factorA numerical factor that determines how strongly future rewards and values are weighted relative to immediate rewards.
- reward engineeringThe process of specifying how outcomes or behaviors are evaluated and mapped to reward feedback.
- stochastic policyA stochastic policy is a decision rule that assigns a probability distribution over possible actions for each state.
- state transitionA state transition is a change from one environment state to another resulting from an action and possibly random effects.
- workforce shift schedulingWorkforce scheduling is the assignment of staff levels and working periods to meet anticipated demand while balancing labor cost and service requirements.
- data leakageThe unintended use of information in a model or evaluation that would not legitimately be available when the model is used.
- probability distributionA probability distribution specifies the probabilities assigned to the possible values of a random variable.
- DynamicPricingEnvDynamicPricingEnv is a book-specific simulation environment for pricing a perishable product over discrete time periods with finite inventory.
- demand functionA demand function maps relevant conditions such as price and time to the quantity of a product that customers are expected to buy.
- inventory constraintAn inventory constraint limits sales or actions to the finite quantity of products currently available.
- limited selling windowA limited selling window is a fixed period during which inventory can be sold before sales lose relevance or become impossible.
- perishabilityPerishability is the property of losing product value when an item remains unsold beyond its usable or marketable period.
- perishable productA perishable product is a product whose value declines substantially or becomes zero after a limited shelf life or deadline.
- price-sensitive demandPrice-sensitive demand is demand whose quantity changes in response to the price of the product.
- revenue managementRevenue management is the practice of optimizing prices and inventory availability to maximize revenue over time.
- evaluationThe assessment of a system's performance using evidence and criteria intended to distinguish effective behavior from misleading results.
- offline evaluationOffline evaluation estimates an agent's performance by replaying previously collected interaction data instead of interacting with a live environment.
- signal fidelitySignal fidelity is the degree to which a producer's signal matches the direction of a reference signal.
- supplier lead timeSupplier lead time is the elapsed time between placing an order with a supplier and receiving it.
- 3d bin packing problemThe 3D bin packing problem is a combinatorial optimization problem that places three-dimensional items into containers while respecting geometric constraints and optimizing space usage.
- 3d gridA 3D grid is a discrete array indexed along three spatial dimensions for representing occupied or empty container space.
- PackingEnvPackingEnv is a custom simulation environment that models sequential three-dimensional item placement in trailers.
- trailer loading optimizationTrailer loading optimization is the process of deciding how to place items in trailers to improve capacity usage and reduce logistics cost.
- combinatorial optimizationCombinatorial optimization is optimization over a discrete set of possible configurations or decisions.
- observationAn observation is the information supplied by an environment that represents the current situation available to an agent.
- orientationAn orientation is a particular rotational arrangement of a three-dimensional item for placement in a container.
- packed volumePacked volume is the total three-dimensional volume of items successfully placed in containers.
- space utilizationSpace utilization is the proportion or amount of available container space occupied by successfully packed items.
- gradient ascentAn iterative optimization method that updates parameters in the direction of a function's gradient to increase its value.
- feature extractionFeature extraction is the transformation of raw state information into numerical features that represent properties relevant to prediction.
- feature vectorA feature vector is an ordered collection of numerical features representing an input state or state-action pair.
- linear function approximationA function-approximation technique that estimates values as weighted sums of state or state-action features.
- bias-variance tradeoffThe bias-variance tradeoff is the balance between systematic estimation error and variability across samples or outcomes.
- positional embeddingA representation added to inputs to encode their positions or identities so attention models can use ordering or spatial relationships.
- neural networkA neural network is a parameterized computational model composed of interconnected transformations that map inputs to outputs.
- attentionAn attention mechanism computes input-dependent weights to combine representations according to their relevance to one another.
- contractA fixed, mutually agreed interface that connected software modules are not allowed to change silently.
- reward hackingReward hacking is the exploitation of a specified reward function in ways that increase its numerical value without achieving the intended objective.
- backpropagationBackpropagation computes gradients of a model loss with respect to parameters by applying the chain rule backward through a computational graph.
- categorical distributionA categorical distribution assigns probabilities to a finite set of mutually exclusive outcomes.
- reward-to-goReturn-to-go is the discounted sum of rewards from a specified decision time onward in a trajectory.
- standard deviationStandard deviation is the square root of variance and measures the typical spread of a probability distribution.
- backpropagation phaseThe backpropagation phase of Monte Carlo tree search propagates a simulation outcome toward the root while updating node statistics.
- expansion phaseThe expansion phase is the Monte Carlo tree search step that adds a child node for an action not previously represented at the selected node.
- selection phaseThe selection phase is the Monte Carlo tree search step that follows a path from the root through already expanded nodes to a node eligible for expansion or termination.
- simulation phaseThe simulation phase is the Monte Carlo tree search step that rolls out actions from an expanded node to obtain an outcome estimate.
- low-rank adaptationLoRA is a parameter-efficient fine-tuning technique that trains low-rank updates to a model's weights while keeping its original weights fixed.
- breadth-first searchBreadth-first search explores graph or tree nodes in order of their distance from a starting node, typically using a FIFO queue.
- depth-first searchDepth-first search explores a graph or tree by following one branch as far as possible before backtracking, typically using a stack or recursion.
- criticThe value-estimating component of an actor-critic system that predicts expected returns and supplies a learning baseline.
- model-based approachA model-based method solves a decision problem by using a known or learned model of the environment to plan ahead.
- accumulating traceAn accumulating trace increases by a fixed increment when its associated state or state-action pair is visited, in addition to retaining its decayed value.
- n-step TDn-step TD is a temporal-difference method whose target uses rewards and state estimates across n future steps.
- replacing traceA replacing trace sets the trace of a visited state or state-action pair to a fixed maximum value instead of accumulating repeated increments.
- portfolio constructionThe process of choosing investment holdings and their allocations while balancing expected return, risk, diversification, and trading costs.
- portfolio managementThe ongoing process of choosing and adjusting investments to balance expected return, risk, and costs over time.
- signal generationThe process of turning raw information into numeric views about the likely direction or behavior of assets.
- decision variableA decision variable is a quantity whose value is chosen by an optimization model to represent an available decision.
- business abstract layerThe business abstract layer is a problem-framing layer that identifies, scopes, and describes the operational problem and its relevant entities, assumptions, and metrics.
- evaluate and test layerThe evaluate and test layer is the validation layer that checks whether an environment behaves logically and produces trustworthy feedback before training.
- simulation design layerThe simulation design layer is the environment-construction layer that translates a business problem into states, actions, rewards, and transition behavior.
- step functionA step function applies an agent action to an environment and returns the resulting observation, feedback, and episode status.
- format bonusAn additional reward for presenting an answer in a required format, independent of whether its content is correct.
- group-relative baselineA baseline that compares a response’s reward with the rewards of other responses generated for the same prompt.
- Schulman k3 estimatorA nonnegative Monte Carlo estimator of reverse Kullback–Leibler divergence computed from differences between log probabilities.
- vocabularyA language model’s vocabulary is the set of tokens it can represent and select as output.
- residual connectionA neural-network connection that adds a layer's input directly to its transformed output.
- treeA tree is a hierarchical data structure of nodes connected by parent–child relationships without cycles.
- Proximal Policy OptimizationPPO is a policy-gradient reinforcement-learning algorithm that constrains policy updates to maintain stable improvements.
- reward model trainingReward modeling trains a model to estimate the desirability of responses from preference comparisons.
- gradient checkpointingGradient checkpointing reduces neural-network activation memory by recomputing activations during the backward pass instead of retaining them.
- online networkThe currently updated neural network in DQN that estimates action values and supports action selection.
- experienceExperience is a recorded interaction tuple containing a state, action, reward, and next state.
- feature engineeringFeature engineering is the construction or transformation of input features into representations suitable for a model or learning algorithm.
- state designState design is the technique of selecting and structuring state information so that it captures the factors needed to predict relevant future outcomes and make decisions.
- adversarial problemAn adversarial environment is an environment in which other agents or processes can react strategically to the decision maker's actions.
- cold start problemThe cold start problem is the difficulty of making effective recommendations or decisions when a new action, item, user, or system has little or no interaction data.
- combinatorial banditA combinatorial bandit is a bandit model in which each decision selects a combination or set of actions whose joint outcome determines the payoff.
- covariate shiftCovariate shift is a distribution change in which the distribution of input contexts changes while the conditional outcome relationship may remain stable.
- EXP3EXP3 is an adversarial bandit algorithm that uses exponentially weighted action estimates to select actions under potentially adversarial rewards.
- non-stationarityNon-stationarity is a condition in which the statistical relationships governing contexts, actions, or outcomes change over time.
- advantage normalizationAdvantage normalization rescales a batch of advantage estimates to have zero mean and unit variance.
- entropyEntropy quantifies the uncertainty or randomness of a probability distribution.
- deep reinforcement learningA reinforcement-learning approach that uses deep neural networks to approximate policies, value functions, or related decision components.
- optimizationOptimization is the selection of controllable decisions that maximize or minimize an objective subject to constraints.
- warehouse layoutThe spatial arrangement of warehouse cells, obstacles, item locations, and depot locations.
- warehouse order picking environmentA simulated warehouse setting in which an agent navigates to collect ordered items and return them to a depot.
- state encoderA state encoder transforms unstructured input information into structured numerical features for an agent's state representation.
- sparse rewardA sparse reward is a feedback signal provided at only a small number of points, rather than continuously throughout a task.
- behavior policyA behavior policy is the policy that selects actions and generates experience for a learning algorithm.
- fine-tuningFurther training of a pretrained model on data or objectives suited to a particular task or use.
- Sharpe ratioA measure of risk-adjusted investment performance, commonly calculated as excess return divided by return volatility.
- representation layerA component that converts unstructured inputs into a numerical representation suitable for a downstream decision system.
- low-rank subspaceA low-rank subspace is a restricted set of directions whose dimensionality is smaller than that of the full space being represented.
- explorationExploration is the selection of actions whose outcomes are uncertain in order to gather information about their payoffs.
- imitation learningImitation learning trains an agent to reproduce behavior demonstrated by an expert or another policy.
- queryA query is an attention vector representing the information an input element seeks from other elements.
- query-key-value frameworkThe query-key-value framework represents each input with query, key, and value vectors to compute attention-weighted outputs.
- valueA value is an attention vector containing the information contributed by an input element after attention weights are computed.
- Transformer encoder layerA Transformer encoder block containing self-attention and position-wise feedforward sub-layers with residual connections and normalization.
- layer normalizationA normalization technique that rescales and recenters activations within each layer to stabilize neural-network computation.
- permutation invarianceThe property that reordering inputs changes the output only by the corresponding reordering, without changing relational results.
- policy headA policy head is the output component of a policy model that converts an internal representation into action decisions or action parameters.
- position-wise feedforward networkA feedforward neural subnetwork applied independently to each position or entity representation.
- queueA queue is a data structure that processes elements in first-in, first-out order.
- stackA stack is a data structure that processes elements in last-in, first-out order.
- linear modelA linear model represents an outcome as a weighted linear combination of input features.
- credit assignmentCredit assignment is the problem of determining which earlier decisions contributed to a later success or failure.
- ablation studyA method of evaluating a system by removing or disabling one component or input and comparing its effects.
- fixed-width state representationA state representation that keeps a constant number of features even when some features are disabled or contain no information.
- Markov processA Markov process is a stochastic process whose future behavior is conditionally independent of the past given its current state.
- delayed or censored rewardA delayed reward is feedback whose value becomes observable only after a time interval following the associated action.
- exploration floorAn exploration floor is a lower bound on the probability or frequency with which an agent continues trying less-established actions.
- forgettingForgetting is an adaptation technique that reduces the influence of older observations when estimating changing payoff relationships.
- optimistic priorAn optimistic prior assigns favorable initial beliefs to uncertain actions or parameters before sufficient observations are available.
- sliding windowA sliding window is a method that restricts estimation to observations from the most recent fixed or adaptive time interval.
- warm startA warm-start approach initializes a new decision model from prior data, parameters, or a related trained model.
- exploitationExploitation is choosing the action currently believed to produce the greatest reward based on available knowledge.
- decoding strategyDecoding is the procedure that converts a model’s output scores into a selected token or sequence.
- tokenizationTokenization is the process of dividing text into discrete units called tokens.
- embeddingAn embedding is a dense vector representation of an item, such as a token, learned by a model.
- regretRegret is the difference between the reward obtained by an algorithm and the reward it could have obtained by acting optimally over the same decisions.
- sublinear regretSublinear regret is regret whose growth is slower than linearly with the number of decisions, causing average regret to approach zero.
- dense rewardA stepwise reward is feedback assigned at individual decision steps rather than only at the end of an episode or process.
- target policyA target policy is the decision rule whose value or action preferences an off-policy learning algorithm estimates or optimizes.
- activation functionA function that transforms a neural network unit's pre-activation value, typically introducing nonlinearity.
- chain ruleThe chain rule is a calculus rule for differentiating a composition of functions by multiplying derivatives of its successive components.
- deep learningDeep learning is a machine-learning approach that learns hierarchical representations and predictions using neural networks with multiple learned layers.
- forward passA forward pass computes a neural network's output by propagating an input through its layers in order.
- gradientA gradient is a vector of partial derivatives that indicates how a scalar function changes with respect to multiple variables.
- neural network layerA neural network layer is a computational stage that transforms its inputs into outputs using parameters and an activation operation.
- linear transformationA linear transformation maps inputs to weighted combinations of those inputs, possibly with an added bias.
- nonlinear mappingNonlinearity is a mathematical property that allows a model's output to vary in ways that cannot be represented by a single linear transformation.
- optimizerAn optimizer is an algorithm that updates model parameters using an objective function and, commonly, its gradients.
- data pipelineA sequence of processes that collects, transforms, validates, and delivers data for a system or analysis.
- bandit algorithmA bandit algorithm selects among actions and updates its action-selection strategy from observed rewards while balancing exploration and exploitation.
- next-word predictionA language-model training approach that predicts the next token from the preceding sequence.
- simulationSimulation is the execution of a model that reproduces interactions or outcomes of an environment without operating directly in the real environment.
- continuous controlContinuous control is decision-making in which an agent selects real-valued control signals rather than discrete actions.
- constrained banditA constrained bandit is a bandit model in which action selection must satisfy resource, exposure, fairness, or other operational constraints.
- personalizationPersonalization is the adaptation of decisions, recommendations, or treatments to individual users or contexts.
- proxy reward functionA proxy reward is an observable signal that serves as an approximation for a later or less observable desired outcome.
- curse of dimensionalityThe curse of dimensionality is the rapid growth of computational complexity as the number of dimensions or tracked variables increases.
- supervised learningA machine-learning technique that learns mappings from inputs to target outputs using labeled examples.
- nucleus samplingNucleus sampling draws a token from the smallest set of highest-probability tokens whose cumulative probability reaches a chosen threshold.
- value headA language-model value head estimates the expected reward for a response generated by a language model.
- restaurant scheduling problemA finite-horizon decision problem in which a restaurant chooses when to seat parties to maximize expected revenue under uncertain arrivals.
- agent-environment interaction loopThe recurring reinforcement-learning cycle in which an agent observes a state, takes an action, receives feedback, and reaches a new state.
- framingThe process of identifying and defining a problem, its relevant decisions, and whether a particular approach is warranted.
- division of laborA system design pattern in which a language model creates representations while a reinforcement-learning agent makes decisions.
- encoderA model component that transforms input into a representation suitable for downstream processing or decision-making.
- binary treeA binary tree is a tree in which each node has at most two children.
- decision treeA decision tree is a tree-shaped model in which internal nodes represent tests or decisions and leaf nodes represent outcomes.
- game treeA game tree is a tree representing possible states and moves in a strategic game or other sequential decision process.
- nodeA node is an element in a tree or graph that represents a state, decision point, or piece of information.
- discrete action bottleneckThe discrete action bottleneck is the scalability limitation caused by representing continuous or high-dimensional controls as a finite set of discrete actions.
- policy representation advantagePolicy representation advantage is the benefit of directly learning a compact decision rule when the corresponding value function is more complex.
- value-based methodA value-based method learns action or state values and derives the policy by selecting actions with the highest estimated value.
- bike-sharing station rebalancingBike-sharing station rebalancing is the logistical redistribution of bikes among stations to reduce shortages and surpluses.
- inventory replenishmentInventory replenishment is the process of determining when and how much product to reorder for stocked locations.
- production schedulingProduction scheduling is the assignment and sequencing of manufacturing tasks over available machines and time periods.
- vehicle routing problemThe vehicle routing problem is an optimization problem that determines vehicle routes for serving locations, typically while minimizing travel-related cost or distance.
- classical optimization methodA classical optimization method is a mathematical procedure for solving a decision problem defined by objectives and constraints.
- operations researchOperations research is the discipline of applying mathematical models, logic, and optimization methods to support decision-making under constraints.
- long-term planningPlanning that evaluates how present actions affect outcomes over an extended sequence of future decisions.
- discrete event simulationDiscrete event simulation models a system as an ordered sequence of instantaneous events that change its state over time.
- stochastic simulationA stochastic simulation is a computational model that represents randomness and explores the range of possible system behaviors under uncertainty.
- service-level agreementA service level agreement is a formal commitment specifying expected service performance, such as the availability or fulfillment level of a product or service.
- decision cycleA classification of a decision by whether it recurs periodically, occurs once, or occurs occasionally.
- decision dimensionA classification indicating whether a decision concerns one entity or multiple entities.
- decision levelA classification of a decision by its strategic, tactical, or operational scope.
- decision quantifiabilityThe degree to which a decision can be represented and evaluated quantitatively rather than qualitatively.
- sensitivity analysisAn analysis of how changes in model inputs or parameters affect model outputs or objectives.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
