A learning rate controls how strongly an update changes an estimate or model parameter. Think of it as the size of an adjustment after feedback: the error indicates what needs correcting, while the learning rate determines how aggressively to respond. section 6.1 section 8.1
In reinforcement learning, there may be no fixed label giving the correct answer for each decision. Temporal-difference (TD) learning instead uses experience to produce an error signal. With linear function approximation, the update is θ ← θ + αδφ(s, a), where α is the learning rate, δ is the TD error, and φ(s, a) describes the features of a state–action pair. Larger active features receive larger weight changes; similar future situations can be affected by the same update. section 6.1 section 8.1
Different rates for different training stages
A language-model training pipeline uses a learning rate of 2e-5 for supervised fine-tuning, 1e-5 for reward-model training, and 1e-6 for PPO. The rates decrease across stages: PPO makes the smallest updates because it starts from an already useful fine-tuned policy. section 10.8.1
Table 10.1 Key hyperparameters for the RLHF pipeline
|
Component
|
Key setting
|
Value
|
Purpose
|
|
Base model
|
model_name
|
Qwen/Qwen2.5-0.5B
|
Small
enough for single-GPU RLHF
|
|
Sequence
length
|
max_length
|
1024
|
Maximum
prompt-response length
|
|
SFT
|
learning
rate
|
2e-5
|
Moderate
supervised fine-tuning step size
|
|
Reward
model
|
learning
rate
|
1e-5
|
Slower
learning for preference scoring
|
|
Reward
model
|
margin
|
1.0
|
Forces
separation between chosen and rejected responses
|
|
PPO
|
learning
rate
|
1e-6
|
Gentle
policy updates during RL
|
|
PPO
|
clip
epsilon
|
0.2
|
Trust-region-style
clipping
|
|
PPO
|
initial KL
coefficient
|
0.1
|
Penalizes
drift from the SFT model
|
|
PPO
|
target KL
|
0.02
|
Used to
adapt the KL coefficient
|
|
PPO
|
rollout
count
|
32
|
Responses
generated per PPO iteration
|
Compare the stage-specific learning rates alongside the other settings in the RLHF pipeline. section 10.8.1
Checking for updates that are too small
In a LoRA and GRPO run on a 3B instruction-tuned model, learning rates of 1e-6 and 1e-5 produced little KL movement and no accuracy improvement. The small LoRA matrices were changing too little to learn. A practical check is to see whether KL is moving before concluding that the method or model is at fault; the reported range for LoRA with GRPO is roughly 5e-5 to 1e-4. section 11.6.1






















