Response-formatting reward
A response-formatting reward is a reward signal assigned according to whether a model output satisfies a required structural format.
Build a Reasoning Model (From Scratch)
A reward component that gives the model credit for producing outputs in the required tag format, regardless of whether the answer is correct.
- How it is used
- Adds an auxiliary structural signal to GRPO training with verifiable correctness rewards.
- How deep it goes
- Implements ordered tag validation, loss integration, prompt changes, and reward-balance diagnostics, but defers broader GRPO extensions.
- Examples
- Uses <think> tags, tokenizer encodings, malformed-tag tests, updated prompts, and GRPO training metrics.
- What you can do after
- Apply it via a GRPO loss function to reward correctly ordered reasoning tags and monitor whether formatting overwhelms answer correctness.
Read how Build a Reasoning Model (From Scratch) explains response-formatting reward →
Reinforcement Learning from Human Feedback
A format reward is a reward signal assigned according to whether a model output follows a required structural presentation convention.
- How it is used
- Uses a minor reinforcement-learning reward to make reasoning outputs follow predictable structural conventions.
- How deep it goes
- Defines the motivation and gives a tag-based example, but does not specify reward computation, weighting, or optimization details.
- Examples
- Uses <redacted_thinking>...</redacted_thinking> before an answer and names DeepSeek R1, OpenReasonerZero, Magistral, and Skywork OR-1.
- What you can do after
- Operate/configure reasoning-model training with an auxiliary reward for required output structure.
Read how Reinforcement Learning from Human Feedback explains response-formatting reward →
Related concepts
- <think> tagaddresses · A think tag is a markup element that delimits reasoning-style text from a model's final answer text.
- accuracycontrasts with · Accuracy is the degree to which an output, prediction, or measurement agrees with a correct or reference outcome under a specified criterion.
- reward signalproduces · A reward signal is an evaluative feedback value representing the desirability or payoff of an action, outcome, or state.
- reinforcement learning with verifiable rewardsused by · Reinforcement learning with verifiable rewards is a training approach that optimizes a model using rewards derived from objectively checkable outcomes.
- verifiable rewardcontrasts with · A verifiable reward is a reward signal whose correctness can be determined by an objective criterion or procedure.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.