Output layer

Q: How deeply does Reinforcement Learning from Human Feedback cover output layer?

Explains sequence pooling, attention-mask indexing, linear projection, pairwise loss, and per-token variants without covering distributed loader details.

As explained in Reinforcement Learning from Human Feedback

A scalar head is a model output component that maps hidden representations or logits to a single numerical prediction.

An output layer converts a model’s internal representation into the prediction consumed by the next stage. In a scalar reward model, a small linear head maps the final hidden representation of a sequence to one numerical score. Pairwise preference learning then compares the chosen and rejected scores; the loss depends on their difference, not their absolute offset. section 5.1 section 5.9

A typical scalar output pipeline proceeds as follows:

Run tokenized input through the base language model and request hidden states. section 5.2
Select one sequence representation, commonly the last non-padding or EOS token, using the attention mask. section 5.2
Apply a linear layer from the model hidden size to one output unit, producing one scalar per sequence. section 5.2
Compare scalar outputs for chosen and rejected completions with a sigmoid or log-sigmoid preference loss. section 5.1 section 5.2

The same architectural idea supports different prediction granularities. A standard preference reward model emits one logit for an entire sequence, whereas an outcome reward model applies a scalar head at every token and predicts correctness probabilities for completion tokens. Prompt tokens are masked during training and scoring. section 5.4 section 5.9

Minimal scalar head

Given a base language model with hidden size h, define a head as a linear map from h features to one value. Run chosen and rejected inputs separately, obtain rewards_chosen and rewards_rejected, and optimize -logsigmoid(rewards_chosen - rewards_rejected). section 5.2

import torch.nn as nn

rewards_chosen = model(**inputs_chosen) 1
rewards_rejected = model(**inputs_rejected)  2

loss = -nn.functional.logsigmoid(rewards_chosen - rewards_rejected).mean()

The listing connects the scalar output layer directly to pairwise reward-model training. section 5.2

How Reinforcement Learning from Human Feedback covers it

Cover of Reinforcement Learning from Human Feedback

liveBook manning.com

Read Reinforcement Learning from Human Feedback on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

Classic Computer Science Problems in Swift: Essential techniques for practicing programmers
liveBook manning.com

Grokking Machine LearningThe neural-network layer that produces the model's final outputs or predictions.
liveBook manning.com

Inside AIAn output layer is the neural-network layer that produces the model’s final prediction or output.
liveBook manning.com

Grokking AI Algorithms, Second EditionAn output layer is the final neural-network layer whose units produce the network's externally returned results.
liveBook manning.com

Data Without LabelsAn output layer is the final neural-network layer that transforms preceding activations into a prediction or other task output.

Compare how 6 books explain output layer →

Related concepts

deep learningpart of · Deep learning is a machine-learning approach that uses neural networks with multiple computational layers to learn representations from data.
neural network layerkind of · A neural network layer is a computational stage or component that transforms input representations into outputs within a neural network.
language modelpart of · A language model is a computational model that represents patterns in linguistic tokens to estimate their probabilities and predict or generate sequences of them.
linear layerimplemented by · A linear layer is a neural-network layer that maps input features to outputs by a learned affine transformation, typically connecting each output to every input feature.
transformer architecturepart of · A transformer architecture is a neural-network architecture that processes sequences or other structured inputs using attention mechanisms and related transformations.
embedding layerequivalent to · An embedding layer maps discrete identifiers, such as tokens or categories, to learned dense numerical vectors.

This explainer was generated by Manning's knowledge graph from the text of Reinforcement Learning from Human Feedback. The section links point to the parts of the book it draws on.