chapter twelve

12 Building Reinforcement Learning systems: LLM-guided RL agent for portfolio management

 

This chapter covers

  • Why a working reinforcement learning system is mostly engineering, not algorithm
  • A practical framework for designing real-world RL systems
  • Integrating LLM fine-tuning into an RL system
  • Building, evaluating, and iterating an end-to-end LLM-guided trading system
In theory, there is no difference between theory and practice. In practice, there is.

Benjamin Brewster, American industrialist

Over the previous four chapters you have built a Deep Q-Network, a REINFORCE agent with a Transformer policy, a full PPO-based RLHF pipeline, and a GRPO fine-tuner with LoRA and verifiable rewards. You understand the algorithms. You could re-derive the policy gradient theorem on a whiteboard and explain why PPO clips its ratio. And now someone hands you a real problem — manage a portfolio, route a fleet, cool a building, align a model — and asks you to build something that actually works, that they can run every day, that will not embarrass them the first time reality deviates from the slide deck.

12.1 The common structure of deep reinforcement learning algorithms

12.2 The gap between an algorithm and a system

12.3 The capstone: an LLM-guided RL portfolio manager

12.4 Framing and engineering the environment

12.5 Representation engineering with a language model

12.6 Wiring the model into the loop

12.7 Designing the evaluation

12.8 Results: does a better representation build a better system?

12.9 Design principles for applied RL systems

12.9.1 Representation has the greatest impact

12.9.2 The reward is a specification the agent will exploit

12.9.3 Guard the boundary of time: never let the future leak in

12.9.4 The agent matters less than you think

12.9.5 Evaluate as if you are trying to prove yourself wrong

12.10 The pattern beyond trading

12.11 Summary