Chapter 12: Building Reinforcement Learning systems: LLM-guided RL agent for portfolio management
This introduction frames practical RL as a systems-engineering challenge and illustrates it with an LLM-guided portfolio manager.
After this chapter you will be able to
- RL systems engineering risks. Explain why a working RL system depends on more than its algorithm, using state representation and reward design as examples.
- Portfolio decision and encoding loop. Trace the portfolio manager's daily decision at the level described, and distinguish the language model's role from the RL agent's role.
- Shared Deep RL Loop. Trace the common deep RL training loop and explain how it helps compare algorithms.
- Learned Network Representations. Distinguish the value, policy, and actor–critic functions a deep RL network can represent, using examples from the section.
- On-Policy and Off-Policy Experience. Distinguish on-policy and off-policy experience sources and describe how DQN and the on-policy methods use their data.
- Learning Target Variations. Explain the role of a learning target and compare the targets used by DQN, REINFORCE, PPO, and GRPO.
- Differentiable Learning Objectives. Describe how an objective turns a target into a differentiable quantity, contrasting DQN's loss with policy-gradient objectives.
- Updates and Training Stabilizers. Explain the role of the gradient update and stabilizers, using examples from the algorithms discussed.
- Algorithm and system gap. Explain why a correct RL algorithm is only one component of a deployable system and where most design effort and risk lie.
- RL problem framing. State the conditions the section gives for deciding whether a problem justifies reinforcement learning.
- Environment fidelity and learning. Explain why the environment is treated as a core product in an applied RL system.
- State representation and visibility. Explain why state representation matters and describe the language model's role in this chapter.
- Reward specification risks. Explain how the written reward can produce behavior different from the designer's intention.
- Evaluation and misleading evidence. Explain why evaluation must distinguish genuine performance from misleading evidence when guiding iteration.
- Bounded portfolio decision. Describe the capstone’s daily allocation decision and explain why the problem is kept narrow.
- Sequential portfolio management rationale. Explain why the section treats portfolio construction as a sequential reinforcement learning problem rather than a one-shot optimization.
- Modular daily decision loop. Trace how news becomes an allocation and reward, and explain the benefit of the numeric interface between the language model and agent.
- Oracle student distillation. Explain how the expensive oracle’s news signals are distilled into a deployable student and why this is useful.
- Constructed market evaluation. Explain what the oracle-defined market lets the experiment measure and what the results do not claim.
- Readable constrained portfolio actions. Explain why the agent selects among eight named portfolio templates rather than producing continuous asset weights, and how the templates enforce the position cap.
- Fixed width state ablation. Describe the four state blocks and explain how the news switch enables a cleaner comparison across modes.
- Leakage resistant step ordering. Explain how the environment prevents future returns from influencing the portfolio decision.
- Shaped daily reward. Describe how the daily reward combines portfolio return with costs and risk-related penalties.
- Reward term scale balance. Explain why reward-term scales must be checked against the agent's actual portfolio behavior, using the cash-heavy failure as an example.
- Numeric news interface contract. Explain how the representation layer converts news into a numeric state block and why a stable MarketSignal schema lets different producers feed the same RL agent.
- Cost-aware labeling pipeline. Describe how the news-to-signal pipeline controls labeling cost and makes oracle labeling scalable and resumable.
- Calibrated signal training targets. Explain what makes an oracle signal a useful training target and how the labeling process promotes that quality.
- Answer-only fine-tuning setup. Describe the training example and how supervised fine-tuning and LoRA are reused to teach the model to output MarketSignal JSON.
- Signal Driven Market Construction. Explain how the adapter and oracle signal make the synthetic market respond to news.
- Oracle Approximation And Trading. Explain the experiment's expectation about signal quality and trading performance.
- Mode Specific News Blocks. Describe how each mode receives its own signal while sharing the same market prices.
- Fair Risk-Aware Evaluation. Describe how the five modes are compared on unseen history and why the evaluation reports more than bare return.
- Oracle Signal Fidelity. Explain how the evaluation measures signal fidelity against the oracle and why this middle link matters to the chapter's causal argument.
- Suspiciously Identical Results. Trace how identical results across different signals exposed a problem and describe the diagnosed causes and response.
- Comparative Portfolio Results. Describe how the modes were compared and how the baseline, news-blind, rule-based, off-the-shelf, and fine-tuned results differed.
- Signal Fidelity And Performance. Explain the measured link between signal fidelity and trading performance, including why the off-the-shelf model did not decisively beat the rule.
- Measured Iteration Across Links. Explain how the second iteration checked whether improvements followed the expected signal-to-performance chain.
- Synthetic Market Interpretation. Explain why the reported Sharpe ratios are not forecasts of live-market performance and what general lesson the experiment can still support.
- State Representation Priority. Explain why an underperforming RL system should first be examined for what its agent can observe.
- Language Model Representation Role. Explain how the fine-tuned language model makes financial news usable by the RL agent, and distinguish its role from choosing trades.
- Reward specification and exploitation. Explain why an agent can earn a high reward while behaving contrary to the designer's intentions.
- Relative reward scales. Explain how excessive risk penalties led the portfolio policy to hold cash, and how rebalancing the same reward terms changed its behavior.
- Reward design failure defenses. Describe the three defenses against reward exploitation: simplify the reward, check the scale of individual terms, and inspect the resulting behavior rather than relying on the reward score.
- Look-ahead bias and consequences. Explain how leakage can make a backtest look successful while producing misleading expectations for live performance.
- Decision-time environment ordering. Describe how the portfolio environment enforces the boundary between information available for an allocation and the return that follows it.
- Chronological training and evaluation. Explain why the portfolio backtest trains on earlier history and evaluates on a later unseen period.
- Agent choice versus system design. Explain why applied RL projects should favor a simple, robust algorithm matched to the action space rather than overinvesting in finding or tuning the perfect agent.
- Skeptical evaluation and baselines. Explain why evaluation should try to expose system weaknesses and compare performance against a genuinely challenging baseline.
- Controlled representation comparison. Describe how the capstone isolates the effect of state representation by changing one variable while keeping the other evaluation conditions identical.
- Aggregated risk-aware performance. Explain why performance should be aggregated across episodes and reported with risk and cost measures rather than relying on a single run or bare return.
- Causal chain measurement. Explain why the evaluation should directly verify that the representation improved when claiming it produced a better policy.
- Representation and decision firewall. Explain how a fine-tuned model and an RL agent divide responsibility when unstructured inputs must inform decisions, using an example beyond trading.
- Offline oracle student distillation. Explain why the pattern uses an expensive oracle offline to teach a smaller model that runs in the decision loop.
What it builds on
- DQN theory: Q-learning meets neural networks (chapter 8) · DQN as a value-based deep RL method
- Why optimizing the policy directly? (chapter 9) · Direct policy optimization
- Proximal Policy Optimization: From REINFORCE to PPO (chapter 10) · PPO as a policy-based method
- Off policy vs on policy learning (chapter 6) · The distinction between on-policy and off-policy learning
- The REINFORCE algorithm (chapter 9) · REINFORCE and returns
- GRPO: group relative policy optimization (chapter 11) · GRPO and group-relative learning
- The mathematics of policy gradients (chapter 9) · Policy-gradient objectives
- Training stability in GRPO (chapter 11) · GRPO training stability
Section by section
Introduction
This introduction frames practical RL as a systems-engineering challenge and illustrates it with an LLM-guided portfolio manager.
- RL systems engineering risks: Explain why a working RL system depends on more than its algorithm, using state representation and reward design as examples.
Why it matters: A sound algorithm alone does not ensure a useful system. - Portfolio decision and encoding loop: Trace the portfolio manager's daily decision at the level described, and distinguish the language model's role from the RL agent's role.
Why it matters: The example shows how a language model can supply information to an RL decision-maker without replacing it. - Historical trading strategy backtests: Describe what a backtest is and why the portfolio example makes system mistakes consequential.
Why it matters: Historical testing is a way to examine a strategy, but the example makes clear that misleading tests can have real consequences.
Key terms: state, reward, engineering, state encoder, financial headlines, allocations, RL loop, backtest, historical market data
12.1 The common structure of deep reinforcement learning algorithms
This section identifies the five shared components and training loop underlying deep reinforcement learning algorithms, showing how algorithms differ in their choices of components.
- Shared Deep RL Loop: Trace the common deep RL training loop and explain how it helps compare algorithms.
Why it matters: The shared loop makes an unfamiliar algorithm easier to understand by identifying which part it changes. - Learned Network Representations: Distinguish the value, policy, and actor–critic functions a deep RL network can represent, using examples from the section.
Why it matters: What the network represents determines the kind of prediction the algorithm learns. - On-Policy and Off-Policy Experience: Distinguish on-policy and off-policy experience sources and describe how DQN and the on-policy methods use their data.
Why it matters: The experience source determines whether an algorithm can reuse old data or must continually collect fresh samples. - Learning Target Variations: Explain the role of a learning target and compare the targets used by DQN, REINFORCE, PPO, and GRPO.
Why it matters: The target supplies the learning signal that tells the algorithm what to improve. - Differentiable Learning Objectives: Describe how an objective turns a target into a differentiable quantity, contrasting DQN's loss with policy-gradient objectives.
Why it matters: The objective translates the learning signal into a quantity the network can optimize. - Updates and Training Stabilizers: Explain the role of the gradient update and stabilizers, using examples from the algorithms discussed.
Why it matters: Stabilizers are a major contributor to an algorithm's practical reliability, not just optional decoration.
Key terms: five parts, loop, experience, target, loss, gradient step, value function, policy, actor–critic, DQN
12.2 The gap between an algorithm and a system
This section explains why building a successful RL system requires decisions about framing, environment, representation, reward, and evaluation beyond choosing an algorithm.
- Algorithm and system gap: Explain why a correct RL algorithm is only one component of a deployable system and where most design effort and risk lie.
Why it matters: A working algorithm does not by itself make a reliable real-world system. - RL problem framing: State the conditions the section gives for deciding whether a problem justifies reinforcement learning.
Why it matters: Framing first helps avoid applying RL to a problem that does not need it. - Environment fidelity and learning: Explain why the environment is treated as a core product in an applied RL system.
Why it matters: An agent learns from the environment it is given, so that environment must reflect the problem it is meant to address. - State representation and visibility: Explain why state representation matters and describe the language model's role in this chapter.
Why it matters: Information cannot guide the agent unless it is represented in a form the agent can use. - Reward specification risks: Explain how the written reward can produce behavior different from the designer's intention.
Why it matters: A reward that sounds reasonable can still incentivize behavior that defeats the business objective. - Evaluation and misleading evidence: Explain why evaluation must distinguish genuine performance from misleading evidence when guiding iteration.
Why it matters: Misleading evaluation can make a failing system look successful and send further development in the wrong direction.
Key terms: algorithm, system, problem framing, environment engineering, state representation, reward design, sequential decisions, delayed feedback, generalize, learning from consequences
12.3 The capstone: an LLM-guided RL portfolio manager
This section frames a bounded portfolio-management system that uses distilled news signals to guide reinforcement learning decisions and evaluates the resulting signal’s decision value.
- Systematic trading pipeline: Describe the three stages of a systematic trading strategy and identify which stages the capstone includes.
Why it matters: This places the capstone within the broader trading workflow and makes its execution assumption clear. - Bounded portfolio decision: Describe the capstone’s daily allocation decision and explain why the problem is kept narrow.
Why it matters: A clear boundary makes the portfolio problem manageable and its results easier to interpret. - Sequential portfolio management rationale: Explain why the section treats portfolio construction as a sequential reinforcement learning problem rather than a one-shot optimization.
Why it matters: The sequential nature of the decision explains why the capstone uses an RL policy. - Modular daily decision loop: Trace how news becomes an allocation and reward, and explain the benefit of the numeric interface between the language model and agent.
Why it matters: This shows how an LLM can improve the agent’s representation without making the portfolio decision itself. - Oracle student distillation: Explain how the expensive oracle’s news signals are distilled into a deployable student and why this is useful.
Why it matters: Distillation makes the signal-extraction approach more practical for a live decision loop. - Constructed market evaluation: Explain what the oracle-defined market lets the experiment measure and what the results do not claim.
Why it matters: This distinction prevents a controlled teaching experiment from being mistaken for evidence of live trading profits.
Key terms: signal generation, alpha research, portfolio construction, execution, closing price, Business Abstract Layer, allocation, ETFs, cash, policy
12.4 Framing and engineering the environment
This section designs a portfolio environment with readable actions, an ablatable state, leakage-resistant steps, balanced rewards, and comparable baselines.
- Readable constrained portfolio actions: Explain why the agent selects among eight named portfolio templates rather than producing continuous asset weights, and how the templates enforce the position cap.
Why it matters: The action space makes portfolio decisions understandable while enforcing a constraint by design. - Fixed width state ablation: Describe the four state blocks and explain how the news switch enables a cleaner comparison across modes.
Why it matters: A fixed representation lets the comparison isolate the contribution of the news signal without changing the network architecture. - Leakage resistant step ordering: Explain how the environment prevents future returns from influencing the portfolio decision.
Why it matters: Keeping future returns inaccessible when the decision is made prevents an unrealistically informed agent. - Shared mechanics and baselines: Explain how shared environment mechanics and simple baselines support a fair evaluation.
Why it matters: Shared calculations and useful baselines make it easier to trust a performance comparison. - Shaped daily reward: Describe how the daily reward combines portfolio return with costs and risk-related penalties.
Why it matters: The reward specifies how the system balances earning returns against trading costs and undesirable portfolio behavior. - Reward term scale balance: Explain why reward-term scales must be checked against the agent's actual portfolio behavior, using the cash-heavy failure as an example.
Why it matters: A reward can improve while the agent learns behavior that defeats the real investment objective.
Key terms: action space, allocation templates, position cap, categorical-policy PPO, state, market price features, news/LLM block, portfolio features, execution features, ablation
12.5 Representation engineering with a language model
This section designs a stable numeric news interface and builds and fine-tunes a language-model extractor to supply it to a portfolio RL agent.
- Numeric news interface contract: Explain how the representation layer converts news into a numeric state block and why a stable MarketSignal schema lets different producers feed the same RL agent.
Why it matters: A stable interface lets the system benefit from news without tying the agent to one particular extractor. - Cost-aware labeling pipeline: Describe how the news-to-signal pipeline controls labeling cost and makes oracle labeling scalable and resumable.
Why it matters: Good data preparation saves money while making a large labeling run practical to complete and extend. - Calibrated signal training targets: Explain what makes an oracle signal a useful training target and how the labeling process promotes that quality.
Why it matters: A model can learn meaningful news signals only if its targets are calibrated and usable by the environment. - Answer-only fine-tuning setup: Describe the training example and how supervised fine-tuning and LoRA are reused to teach the model to output MarketSignal JSON.
Why it matters: Reusing the existing fine-tuning methods makes the new signal extractor and its base-model comparison practical.
Key terms: representation layer, news block, signal schema, MarketSignal, API, oracle, curate, label, pilot, example identity
12.6 Wiring the model into the loop
This section connects news signals to a synthetic market and agent observations to compare portfolio-management modes reproducibly.
- Signal Driven Market Construction: Explain how the adapter and oracle signal make the synthetic market respond to news.
Why it matters: Signal quality can matter in the comparison only if the market responds to the signal. - Oracle Approximation And Trading: Explain the experiment's expectation about signal quality and trading performance.
Why it matters: This connects the quality of the news representation to the performance the experiment tests. - Mode Specific News Blocks: Describe how each mode receives its own signal while sharing the same market prices.
Why it matters: Sharing the market while changing the news block makes the signal comparison meaningful. - PPO Training And Randomness: Describe the agent training choices that keep the comparison focused on the signal.
Why it matters: Consistent agent training helps distinguish signal effects from differences in exploration or random draws.
Key terms: signal_to_asset_effect, theme-to-ETF mapping, oracle signal, generate_signal_driven_market, distillation, news-blind mode, rule-based signal, base model, fine-tuned model, news block
12.7 Designing the evaluation
This section designs a fair portfolio evaluation, measures signal fidelity, and shows how suspiciously identical results exposed a broken system.
- Fair Risk-Aware Evaluation: Describe how the five modes are compared on unseen history and why the evaluation reports more than bare return.
Why it matters: A fair, risk-aware comparison helps distinguish a usable strategy from a lucky or reckless one. - Oracle Signal Fidelity: Explain how the evaluation measures signal fidelity against the oracle and why this middle link matters to the chapter's causal argument.
Why it matters: Measuring signal quality helps explain a portfolio result rather than merely reporting it. - Suspiciously Identical Results: Trace how identical results across different signals exposed a problem and describe the diagnosed causes and response.
Why it matters: An unusually clean result can reveal that the system is not using the information it was built to exploit.
Key terms: held-out test window, transaction costs, Sharpe ratio, worst drawdown, turnover, signal fidelity, oracle, mean error, direction, signal matrices
12.8 Results: does a better representation build a better system?
This section interprets the portfolio experiment’s results, links signal fidelity to trading performance, and explains the limits of the findings.
- Portfolio Performance Measures: Describe what Sharpe ratio, annualized return and volatility, maximum drawdown, turnover, and final value reveal about a portfolio.
Why it matters: These measures let you distinguish a profitable result from one achieved with excessive risk or trading. - Comparative Portfolio Results: Describe how the modes were compared and how the baseline, news-blind, rule-based, off-the-shelf, and fine-tuned results differed.
Why it matters: The comparison shows that adding a language model is not enough on its own; the quality of its contribution matters. - Signal Fidelity And Performance: Explain the measured link between signal fidelity and trading performance, including why the off-the-shelf model did not decisively beat the rule.
Why it matters: Measuring the intermediate signal helps explain why a representation improves decisions rather than merely showing that the final score changed. - Measured Iteration Across Links: Explain how the second iteration checked whether improvements followed the expected signal-to-performance chain.
Why it matters: Checking intermediate changes helps reveal gains that might otherwise come from an unexplained bug or leak. - Synthetic Market Interpretation: Explain why the reported Sharpe ratios are not forecasts of live-market performance and what general lesson the experiment can still support.
Why it matters: Separating a clean demonstration from a live-market forecast prevents overclaiming what the experiment proves.
Key terms: Sharpe ratio, annualized return, annualized volatility, maximum drawdown, turnover, final value, test window, news-blind RL agent, buy-and-hold, equal-weight
12.9 Design principles for applied RL systems
This section introduces the design principles that follow and their connection to concrete engineering decisions.
12.9.1 Representation has the greatest impact
This section argues that state representation should be the first priority in improving an RL system and illustrates the principle with language-model-derived market context.
- State Representation Priority: Explain why an underperforming RL system should first be examined for what its agent can observe.
Why it matters: This helps you diagnose an underperforming system before spending effort tuning its agent. - Capstone Representation Comparison: Describe what the capstone changes and what it holds fixed to isolate the role of representation.
Why it matters: This comparison makes representation the lever being examined rather than confounding it with other system changes. - Language Model Representation Role: Explain how the fine-tuned language model makes financial news usable by the RL agent, and distinguish its role from choosing trades.
Why it matters: This clarifies how a language model can supply useful information without taking over the agent's decision.
Key terms: state, state representation, policy, market context, environment, reward, agent, evaluation, language model, representation
12.9.2 The reward is a specification the agent will exploit
This section explains how reward specifications can silently induce undesirable portfolio behavior and how to detect and prevent such failures.
- Reward specification and exploitation: Explain why an agent can earn a high reward while behaving contrary to the designer's intentions.
Why it matters: A good reward score is not proof that the agent is doing what you intended. - Relative reward scales: Explain how excessive risk penalties led the portfolio policy to hold cash, and how rebalancing the same reward terms changed its behavior.
Why it matters: Even sensible reward terms can produce a useless policy when their relative weights are wrong. - Reward design failure defenses: Describe the three defenses against reward exploitation: simplify the reward, check the scale of individual terms, and inspect the resulting behavior rather than relying on the reward score.
Why it matters: These checks can reveal silent reward failures that a reasonable-looking training curve misses.
Key terms: reward design, reward-hacking, portfolio return, transaction costs, drawdown, volatility, risk penalties, reward coefficients, reward terms, relative sizes
12.9.3 Guard the boundary of time: never let the future leak in
This section explains look-ahead bias and how to prevent it through decision-time ordering and chronological evaluation.
- Look-ahead bias and consequences: Explain how leakage can make a backtest look successful while producing misleading expectations for live performance.
Why it matters: A strong backtest can be an illusion if the agent had access to the future. - Decision-time environment ordering: Describe how the portfolio environment enforces the boundary between information available for an allocation and the return that follows it.
Why it matters: Enforcing the order in the environment prevents the agent from using outcomes it could not yet know. - Chronological training and evaluation: Explain why the portfolio backtest trains on earlier history and evaluates on a later unseen period.
Why it matters: The evaluation should test decisions on a later period that did not influence training.
Key terms: leakage, look-ahead bias, backtest, step function, state, allocation, news signal, chronological split, training, evaluation
12.9.4 The agent matters less than you think
This section argues for using a sound, appropriately matched RL agent while prioritizing representation and evaluation over excessive algorithm tuning.
- Agent choice versus system design: Explain why applied RL projects should favor a simple, robust algorithm matched to the action space rather than overinvesting in finding or tuning the perfect agent.
Why it matters: This helps you spend effort on the parts of an RL system most likely to change its outcome. - Capstone agent and action space: Describe how the capstone's portfolio choices and reward fit its choice of PPO with a categorical policy.
Why it matters: The capstone illustrates how an existing agent can be matched to a problem without inventing a specialized algorithm. - Exploration during agent training: Explain why maintaining exploration mattered for the capstone agent during training.
Why it matters: Even a simple, stable agent needs enough exploration to avoid settling prematurely on one action.
Key terms: representation, evaluation, action space, PPO, categorical policy, discrete portfolio choices, dense daily reward, exploration, policy collapse
12.9.5 Evaluate as if you are trying to prove yourself wrong
This section explains how to design skeptical evaluations that test whether an RL system’s apparent improvement is meaningful and attributable to its cause.
- Skeptical evaluation and baselines: Explain why evaluation should try to expose system weaknesses and compare performance against a genuinely challenging baseline.
Why it matters: A skeptical test against a hard baseline helps distinguish useful performance from a misleading result. - Controlled representation comparison: Describe how the capstone isolates the effect of state representation by changing one variable while keeping the other evaluation conditions identical.
Why it matters: Controlling the comparison makes the source of a performance difference easier to identify. - Aggregated risk-aware performance: Explain why performance should be aggregated across episodes and reported with risk and cost measures rather than relying on a single run or bare return.
Why it matters: Repeated, risk-aware measurement makes it harder to mistake luck or excessive risk for a strong strategy. - Causal chain measurement: Explain why the evaluation should directly verify that the representation improved when claiming it produced a better policy.
Why it matters: Checking the intermediate link strengthens the explanation for an observed improvement.
Key terms: evaluation, baselines, state representation, evaluation window, episodes, return per unit of risk, worst drawdown, trading cost, bare return, causal chain
12.10 The pattern beyond trading
This section generalizes the portfolio system’s LLM-and-RL design pattern to other domains and emphasizes designing and evaluating the whole system.
- Representation and decision firewall: Explain how a fine-tuned model and an RL agent divide responsibility when unstructured inputs must inform decisions, using an example beyond trading.
Why it matters: This division lets the agent use information it cannot directly read without handing its decisions to the language model. - Offline oracle student distillation: Explain why the pattern uses an expensive oracle offline to teach a smaller model that runs in the decision loop.
Why it matters: Distillation can make useful encoding practical across the many calls an RL system requires. - Pure LLM decision limitations: Describe the section’s reasons for caution about using an LLM alone for large-scale applied decision-making.
Why it matters: The caution explains why the section assigns decisions to RL rather than assuming an LLM can handle the whole optimization task. - Whole system design: Explain why building an applied RL system requires more than choosing an algorithm.
Why it matters: A good algorithm alone does not settle how the problem is represented or whether the resulting system works.
Key terms: structured state, firewall, encoding fidelity, contract, distillation, oracle, student, frontier model, sequential decision-making, delayed reward
12.11 Summary
Concepts explained in this chapter
- stateA state describes the relevant condition of an environment at a particular point in a sequential decision process.
- data leakageThe unintended use of information in a model or evaluation that would not legitimately be available when the model is used.
- distillationA model-training technique in which a student model learns to reproduce useful outputs or behavior of a teacher model.
- contractA fixed, mutually agreed interface that connected software modules are not allowed to change silently.
- look-ahead biasBias caused by using information that would not have been available at the time of a decision or prediction.
- evaluationThe assessment of a system's performance using evidence and criteria intended to distinguish effective behavior from misleading results.
- signal fidelitySignal fidelity is the degree to which a producer's signal matches the direction of a reference signal.
- data pipelineA sequence of processes that collects, transforms, validates, and delivers data for a system or analysis.
- large language modelA machine-learning model trained on large text corpora to model and generate language.
- portfolio constructionThe process of choosing investment holdings and their allocations while balancing expected return, risk, diversification, and trading costs.
- portfolio managementThe ongoing process of choosing and adjusting investments to balance expected return, risk, and costs over time.
- signal generationThe process of turning raw information into numeric views about the likely direction or behavior of assets.
- state encoderA state encoder transforms unstructured input information into structured numerical features for an agent's state representation.
- Sharpe ratioA measure of risk-adjusted investment performance, commonly calculated as excess return divided by return volatility.
- fine-tuningFurther training of a pretrained model on data or objectives suited to a particular task or use.
- representation layerA component that converts unstructured inputs into a numerical representation suitable for a downstream decision system.
- ablation studyA method of evaluating a system by removing or disabling one component or input and comparing its effects.
- action spaceAn action space is the set or structured specification of actions that an agent can submit to an environment.
- fixed-width state representationA state representation that keeps a constant number of features even when some features are disabled or contain no information.
- objective functionA mathematical quantity that a learning or optimization procedure seeks to minimize or maximize.
- deep reinforcement learningA reinforcement-learning approach that uses deep neural networks to approximate policies, value functions, or related decision components.
- experienceExperience is a recorded interaction tuple containing a state, action, reward, and next state.
- neural networkA neural network is a parameterized computational model composed of interconnected transformations that map inputs to outputs.
- off-policy methodOff-policy learning is reinforcement learning in which a target policy is evaluated or improved using experience generated by a potentially different behavior policy.
- on-policy methodOn-policy learning is reinforcement learning in which the policy generating experience is the same policy being evaluated or improved.
- policyA policy is a decision rule, possibly probabilistic, that assigns actions or action probabilities to states for an agent.
- value functionA value function estimates the expected cumulative return from a state or state-action pair under a specified policy or optimal behavior.
- environmentAn environment is the external system or setting with which an agent interacts and whose responses produce consequences for the agent's actions.
- framingThe process of identifying and defining a problem, its relevant decisions, and whether a particular approach is warranted.
- reinforcement learningReinforcement learning is a machine-learning paradigm in which an agent learns to choose actions through interactions with an environment and feedback about their consequences.
- reward signalA reward is a numerical feedback signal indicating the desirability of an outcome or transition after an agent's action.
- division of laborA system design pattern in which a language model creates representations while a reinforcement-learning agent makes decisions.
- encoderA model component that transforms input into a representation suitable for downstream processing or decision-making.
- reward engineeringThe process of specifying how outcomes or behaviors are evaluated and mapped to reward feedback.
- reward hackingReward hacking is the exploitation of a specified reward function in ways that increase its numerical value without achieving the intended objective.
- replay bufferA replay buffer is a storage structure that retains interaction records for later processing or learning updates.
- AI agentAn entity in reinforcement learning that observes an environment, selects actions, and improves its behavior from feedback.
- explorationExploration is the selection of actions whose outcomes are uncertain in order to gather information about their payoffs.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
