A reward signal indicates how desirable an action or result is. In a Markov decision process, an agent acts in an environment, moves between states, and receives rewards associated with its actions and resulting states. Its policy aims to maximize total rewards, not just the reward from one step. section 5.1.1
From feedback to a training score
On an obstacle course, a dog receives treats for good choices and gentle scolding for mistakes. For language-model training, a reward function instead assigns scores to outputs. One example scores each response by its length plus twice its number of distinct words; preprocessing applies that function to Financial Phrasebank sentences before passing the sentences and scores to trlX. This score rewards length and word diversity—it does not, by itself, assess whether a financial statement is correct. section 5.1.1 section 5.1.2
Listing 5.2 Defining the reward function
def reward_fn(output): length = len(output) diversity = len(set(output.split())) return length + 2 * diversity
The reward function turns two measurable output properties—character length and distinct-word count—into one numerical score. section 5.1.2
Listing 5.4 Training the model with RLHF
default_config = default_ilql_config().to_dict() #1
default_config['train']['tracker'] = None
default_config['train']['batch_size'] = 16
default_config['train']['epochs'] = 20
config = TRLConfig.update(default_config, {})
trainer = trlx.train( #2
model,
samples=samples,
rewards=rewards,
eval_prompts=[
"The S&P has shown",
"The market trends indicate",
"The economic indicators for the quarter are",
"According to recent financial reports"
] * 20,
config=config,
)
The training call supplies samples and their precomputed rewards as separate arguments. The listing uses an ILQL default configuration, despite the surrounding demonstration describing PPO. section 5.1.2
The scoring criterion can change with the task. In a GRPO example, a reward function gives full, partial, or zero reward according to whether a completion has meaningful <think> and <answer> sections. GRPO compares completions within a group, strengthening the learning signal for better relative results without requiring a reward for every token. That makes the final output, rather than each intermediate token, the unit being evaluated. section 5.2.5
Listing 5.22 Basic GRPO training with TRL
from datasets import load_dataset #1
dataset = load_dataset("your_dataset_name", split="train")
def reward_func(completions, **kwargs): #2
pattern = r"<think>(.*?)</think>\s*<answer>(.*?)</answer>"
rewards = []
for completion in completions:
match = re.search(pattern, completion, re.DOTALL)
if match:
think = match.group(1).strip()
answer = match.group(2).strip()
if len(think) > 20 and len(answer) > 0:
rewards.append(1.0)
elif len(think) > 0 or len(answer) > 0:
rewards.append(0.5)
else:
rewards.append(0.0)
else:
rewards.append(0.0)
return rewards
training_args = GRPOConfig( #3
output_dir="output",
num_train_epochs=3,
per_device_train_batch_size=4,
gradient_accumulation_steps=2,
logging_steps=10,
)
trainer = GRPOTrainer( #4
model="your_model", # e.g. "Qwen/Qwen3-4B"
args=training_args,
train_dataset=dataset,
reward_funcs=reward_func,
)
trainer.train()
The GRPO reward function returns a score for each completion using explicit full-, partial-, and zero-reward conditions. section 5.2.5
