liveBook · Manning

Reward model

As explained in Applied Reinforcement Learning

A reward model predicts a numerical score for an output to represent its desirability or alignment with preferences.

A reward model scores an output to estimate how well it matches human preferences. For a language model, it takes a prompt and response and returns a single number. The score gives reinforcement learning a scalable proxy for human judgment without requiring a person to rate every generated answer. section 10.6 section 10.7

Choosing a safer response

For a patient reporting chest pain, raters might prefer an answer that acknowledges the concern, avoids a diagnosis, recommends medical evaluation, and mentions emergency escalation when appropriate. A response that merely lists possible conditions may be less suitable for a support assistant. The reward model learns the preference from examples; the preference is not a rule built into the technique itself. section 10.6

Figure 10.5 Learning a reward model from human preferences.

Follow a query and two candidate responses through a human preference comparison into reward-model training. section 10.6

Training uses pairs of responses to the same query: one chosen and one rejected. A pairwise ranking loss encourages the model to give the chosen response a higher score. In one implementation, a margin is subtracted from the score gap; increasing the gap reduces the loss. section 10.8.4 section 10.7

Comparing scores

With a margin of 1.0, chosen and rejected scores of 2.5 and 1.0 produce a loss of about 0.474. Increasing their gap to 3.0 lowers the loss to about 0.127. Validation accuracy measures how often the chosen response scores higher than the rejected one. section 10.8.4

Listing 10.5 Reward Model Training with Pairwise Ranking Loss
chosen_ids = batch["chosen_input_ids"].to(self.device)
chosen_mask = batch["chosen_attention_mask"].to(self.device)
rejected_ids = batch["rejected_input_ids"].to(self.device)
rejected_mask = batch["rejected_attention_mask"].to(self.device)
 
use_amp = self.device == "cuda" and self.model_config.dtype != torch.float32
with torch.autocast(device_type="cuda", dtype=
➥self.model_config.dtype, enabled=use_amp):
    chosen_rewards = self.reward_model(chosen_ids,
    ➥chosen_mask)  #A
    rejected_rewards = self.reward_model(rejected_ids, 
    ➥rejected_mask)  #B
 
    loss = -F.logsigmoid(chosen_rewards - rejected_rewards –
    ➥self.config.margin).mean()  #C

Inspect the score difference, margin, and pairwise loss in the training logic. section 10.8.4

One implementation starts from a transformer with supervised fine-tuning weights. A reward head maps the last token’s hidden state to one score; causal attention lets that state incorporate the preceding query and response. section 10.8.4

Listing 10.4 Reward Model Architecture
class RewardModel(nn.Module):
    def __init__(
        self,
        base_model: PreTrainedModel,
        tokenizer: PreTrainedTokenizer,
    ):
        super().__init__()
        self.base_model = base_model
        self.tokenizer = tokenizer
 
        hidden_size = base_model.config.hidden_size  #A
 
        self.reward_head = nn.Sequential(
            nn.Linear(hidden_size, hidden_size),
            nn.ReLU(),
            nn.Dropout(0.1),
            nn.Linear(hidden_size, 1),
        )  #B
        self.reward_head.to(dtype=base_model.dtype)  #C
 
    def forward(
        self,
        input_ids: torch.Tensor,
        attention_mask: torch.Tensor,
    ) -> torch.Tensor:
        outputs = self.base_model(
            input_ids=input_ids,
            attention_mask=attention_mask,
            output_hidden_states=True,
        )
 
        hidden_states = outputs.hidden_states[-1]  #D
 
        batch_size = input_ids.shape[0]
        sequence_lengths = attention_mask
        ➥.sum(dim=1) - 1  #E
 
        last_hidden = hidden_states[
            torch.arange(batch_size, device=input_ids.device),
            sequence_lengths,
        ]  #F
 
        reward = self.reward_head(last_hidden)
        ➥.squeeze(-1)  #G
 
        return reward

See the transformer-based reward model and its scalar reward head. section 10.8.4

After reward-model training, a policy generates its own responses and receives scores for PPO fine-tuning. Separate reward-shaping signals can nudge behavior, while a KL penalty helps keep the policy close to the supervised reference. A single score is not a complete evaluation: safety, empathy, urgency, fluency, and domain correctness should also be assessed separately. section 10.7 section 10.8.4 section 10.9

How Applied Reinforcement Learning covers it

How deep it goes
Explains pairwise ranking, a transformer reward head, validation, and PPO integration with core implementation listings, omitting trainer boilerplate.
Examples
Uses chest-pain response comparisons and a headache-query score-gap calculation to illustrate preferences and ranking loss.
What you can do after
Apply reward modeling with PyTorch and Hugging Face Transformers to score preference pairs and support PPO fine-tuning.
How it connects
Connects human preferences with supervised fine-tuning, pairwise ranking loss, PPO, reward shaping, and KL divergence.

Also explained in

Compare how 10 books explain reward model →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.