Proximal policy optimization (PPO) is a reinforcement-learning method for updating a policy without changing it too abruptly. In reinforcement learning from human feedback (RLHF), a language model acts as an agent: each point in generation is a state, adding words or phrases is an action, and feedback supplies rewards that guide its outputs. section 5.1 section 5.1.1
PPO compares an updated policy with the previous one and limits changes that diverge too far. The described clipping interval is [1−ϵ, 1+ϵ], with the adjustment depending on whether the advantage is positive or negative. Smaller updates are intended to make training more stable; a drastic policy change can produce unfavorable behavior that is difficult to correct. section 5.1.1
A reward for financial-text responses
A GPT-2 training workflow uses sentences from Financial Phrasebank and a reward function that scores response length plus twice the number of distinct whitespace-separated words. This is a defined reward signal for the demonstration, not a direct human-preference judgment. section 5.1.2 section 5.1.2
Listing 5.2 Defining the reward function
def reward_fn(output): length = len(output) diversity = len(set(output.split())) return length + 2 * diversity
The reward function makes the training objective concrete: longer outputs and outputs with more distinct words receive higher scores. section 5.1.2
The illustrated trlX workflow prepares samples and rewards, trains a model, and generates text from an evaluation prompt. Check its configuration before treating the listing as a PPO implementation: although PPO is the stated choice, the shown training code starts from `default_ilql_config()`, an implicit language Q-learning configuration. section 5.1.2
Listing 5.4 Training the model with RLHF
default_config = default_ilql_config().to_dict() #1
default_config['train']['tracker'] = None
default_config['train']['batch_size'] = 16
default_config['train']['epochs'] = 20
config = TRLConfig.update(default_config, {})
trainer = trlx.train( #2
model,
samples=samples,
rewards=rewards,
eval_prompts=[
"The S&P has shown",
"The market trends indicate",
"The economic indicators for the quarter are",
"According to recent financial reports"
] * 20,
config=config,
)
The training call shows where samples, rewards, and evaluation prompts enter trlX; its `default_ilql_config()` line is the configuration discrepancy to resolve when selecting PPO. section 5.1.2
