liveBook · Manning

Learning rate

As explained in Applied Reinforcement Learning

A learning rate is a numerical step-size parameter that controls how strongly an update changes an estimate or model parameter.

A learning rate controls how strongly an update changes an estimate or model parameter. Think of it as the size of an adjustment after feedback: the error indicates what needs correcting, while the learning rate determines how aggressively to respond. section 6.1 section 8.1

In reinforcement learning, there may be no fixed label giving the correct answer for each decision. Temporal-difference (TD) learning instead uses experience to produce an error signal. With linear function approximation, the update is θ ← θ + αδφ(s, a), where α is the learning rate, δ is the TD error, and φ(s, a) describes the features of a state–action pair. Larger active features receive larger weight changes; similar future situations can be affected by the same update. section 6.1 section 8.1

Different rates for different training stages

A language-model training pipeline uses a learning rate of 2e-5 for supervised fine-tuning, 1e-5 for reward-model training, and 1e-6 for PPO. The rates decrease across stages: PPO makes the smallest updates because it starts from an already useful fine-tuned policy. section 10.8.1

Table 10.1 Key hyperparameters for the RLHF pipeline
Component
Key setting
Value
Purpose
Base model
model_name
Qwen/Qwen2.5-0.5B
Small enough for single-GPU RLHF
Sequence length
max_length
1024
Maximum prompt-response length
SFT
learning rate
2e-5
Moderate supervised fine-tuning step size
Reward model
learning rate
1e-5
Slower learning for preference scoring
Reward model
margin
1.0
Forces separation between chosen and rejected responses
PPO
learning rate
1e-6
Gentle policy updates during RL
PPO
clip epsilon
0.2
Trust-region-style clipping
PPO
initial KL coefficient
0.1
Penalizes drift from the SFT model
PPO
target KL
0.02
Used to adapt the KL coefficient
PPO
rollout count
32
Responses generated per PPO iteration

Compare the stage-specific learning rates alongside the other settings in the RLHF pipeline. section 10.8.1

Checking for updates that are too small

In a LoRA and GRPO run on a 3B instruction-tuned model, learning rates of 1e-6 and 1e-5 produced little KL movement and no accuracy improvement. The small LoRA matrices were changing too little to learn. A practical check is to see whether KL is moving before concluding that the method or model is at fault; the reported range for LoRA with GRPO is roughly 5e-5 to 1e-4. section 11.6.1

How Applied Reinforcement Learning covers it

How deep it goes
Explains error scaling and a linear TD update, then compares learning rates across fine-tuning stages and diagnostic runs.
Examples
Compares three pipeline rates and reports LoRA and GRPO runs with nearly unchanged KL and no accuracy improvement.
What you can do after
Diagnose insufficient learning by checking KL movement and comparing the learning rate with the update method.
How it connects
Connects temporal-difference learning and linear function approximation to supervised fine-tuning, reward modeling, PPO, and GRPO.

Also explained in

Compare how 23 books explain learning rate →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.