A reward model scores an output to estimate how well it matches human preferences. For a language model, it takes a prompt and response and returns a single number. The score gives reinforcement learning a scalable proxy for human judgment without requiring a person to rate every generated answer. section 10.6 section 10.7
Choosing a safer response
For a patient reporting chest pain, raters might prefer an answer that acknowledges the concern, avoids a diagnosis, recommends medical evaluation, and mentions emergency escalation when appropriate. A response that merely lists possible conditions may be less suitable for a support assistant. The reward model learns the preference from examples; the preference is not a rule built into the technique itself. section 10.6
Figure 10.5 Learning a reward model from human preferences.

Follow a query and two candidate responses through a human preference comparison into reward-model training. section 10.6
Training uses pairs of responses to the same query: one chosen and one rejected. A pairwise ranking loss encourages the model to give the chosen response a higher score. In one implementation, a margin is subtracted from the score gap; increasing the gap reduces the loss. section 10.8.4 section 10.7
Comparing scores
With a margin of 1.0, chosen and rejected scores of 2.5 and 1.0 produce a loss of about 0.474. Increasing their gap to 3.0 lowers the loss to about 0.127. Validation accuracy measures how often the chosen response scores higher than the rejected one. section 10.8.4
Listing 10.5 Reward Model Training with Pairwise Ranking Loss
chosen_ids = batch["chosen_input_ids"].to(self.device)
chosen_mask = batch["chosen_attention_mask"].to(self.device)
rejected_ids = batch["rejected_input_ids"].to(self.device)
rejected_mask = batch["rejected_attention_mask"].to(self.device)
use_amp = self.device == "cuda" and self.model_config.dtype != torch.float32
with torch.autocast(device_type="cuda", dtype=
➥self.model_config.dtype, enabled=use_amp):
chosen_rewards = self.reward_model(chosen_ids,
➥chosen_mask) #A
rejected_rewards = self.reward_model(rejected_ids,
➥rejected_mask) #B
loss = -F.logsigmoid(chosen_rewards - rejected_rewards –
➥self.config.margin).mean() #CInspect the score difference, margin, and pairwise loss in the training logic. section 10.8.4
One implementation starts from a transformer with supervised fine-tuning weights. A reward head maps the last token’s hidden state to one score; causal attention lets that state incorporate the preceding query and response. section 10.8.4
Listing 10.4 Reward Model Architecture
class RewardModel(nn.Module):
def __init__(
self,
base_model: PreTrainedModel,
tokenizer: PreTrainedTokenizer,
):
super().__init__()
self.base_model = base_model
self.tokenizer = tokenizer
hidden_size = base_model.config.hidden_size #A
self.reward_head = nn.Sequential(
nn.Linear(hidden_size, hidden_size),
nn.ReLU(),
nn.Dropout(0.1),
nn.Linear(hidden_size, 1),
) #B
self.reward_head.to(dtype=base_model.dtype) #C
def forward(
self,
input_ids: torch.Tensor,
attention_mask: torch.Tensor,
) -> torch.Tensor:
outputs = self.base_model(
input_ids=input_ids,
attention_mask=attention_mask,
output_hidden_states=True,
)
hidden_states = outputs.hidden_states[-1] #D
batch_size = input_ids.shape[0]
sequence_lengths = attention_mask
➥.sum(dim=1) - 1 #E
last_hidden = hidden_states[
torch.arange(batch_size, device=input_ids.device),
sequence_lengths,
] #F
reward = self.reward_head(last_hidden)
➥.squeeze(-1) #G
return rewardSee the transformer-based reward model and its scalar reward head. section 10.8.4
After reward-model training, a policy generates its own responses and receives scores for PPO fine-tuning. Separate reward-shaping signals can nudge behavior, while a KL penalty helps keep the policy close to the supervised reference. A single score is not a complete evaluation: safety, empathy, urgency, fluency, and domain correctness should also be assessed separately. section 10.7 section 10.8.4 section 10.9









