liveBook · Manning

Reinforcement learning

As explained in LLM Evaluation and Alignment, The Foundational Ideas

Reinforcement learning is a machine-learning paradigm in which an agent learns behavior by optimizing cumulative reward from interactions with an environment.

Reinforcement learning trains an agent through trial-and-error interaction with an environment. The agent generates actions, receives scalar rewards, and updates its policy to increase expected cumulative reward. section 7.3.1

The language-model training loop consists of four parts:

  • Policy: a parameterized model that maps the current state or prompt to an action or response. section 7.3.1
  • Reward: a scalar score for a prompt-response pair; higher scores encourage the policy to produce similar responses more often. section 7.3.1
  • Trajectory: a rollout containing the prompt, generated steps, final response, and observed reward. section 7.3.1
  • Training step: an update that nudges model parameters toward higher expected reward across a batch of trajectories. section 7.3.1

Unlike pretraining and supervised fine-tuning, which learn from next-token targets or demonstrations, reinforcement learning samples multiple candidates and favors those receiving higher rewards. This enables behavior optimization when no single correct demonstration exists. section 7.3

Classical environments can supply rewards directly, as in game scores or checkmate. Language models have no native score for a generated paragraph, so rewards must be engineered through fixed metrics, preference models, or programmatic verifiers. section 7.3.2

Financial assistant

A policy drafts a Q3 earnings summary. A reward function scores faithfulness, source citation, and utility, then an optimizer updates the model so similar prompts are more likely to produce high-scoring summaries. section 7.3.1

Figure 7.2 The reinforcement-learning (RL) loop. The agent learns to make decisions by acting in an environment, observing a state and reward, and updating its policy to maximize cumulative reward.

Visualizes the agent-environment loop: acting, observing state and reward, and updating the policy to maximize cumulative reward. section 7.3

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Defines the policy, reward, trajectory, and update loop, then flags credit assignment without detailing optimizer mechanics.
Examples
Uses financial summarization, Atari, chess, simulated robots, and Pong to illustrate rewards, trajectories, and behavior optimization.
What it assumes
Assumes familiarity with pretraining and supervised fine-tuning while building the reinforcement-learning vocabulary from the policy upward.
What you can do after
Understand how policies learn from scored responses and diagnose why reward design and credit assignment complicate language-model training.
How it connects
Connects reinforcement learning with supervised fine-tuning, reward functions, preference learning, credit assignment, and programmatic verification.

Also explained in

Compare how 28 books explain reinforcement learning →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.