Response-formatting reward

As explained in Build a Reasoning Model (From Scratch)

A reward component that gives the model credit for producing outputs in the required tag format, regardless of whether the answer is correct.

A response-formatting reward adds an auxiliary training signal for required output structure. It can encourage reasoning models to place their reasoning inside <think>...</think> tags alongside an answer-correctness reward. section 7.6 section 7.0

Implement the reward in the training pipeline:

• Use a tokenizer that recognizes <think> and </think>, either by adding special tokens to the base tokenizer or loading a reasoning tokenizer. section 7.6.1
• Inspect only generated tokens after the prompt and return 1.0 when the opening tag occurs before the closing tag; otherwise return 0.0. section 7.6.1
• Add the format score to the correctness reward with a tunable format_reward_weight, and prompt the model to emit the tags explicitly. section 7.6.1 section 7.6.2

Testing malformed tags

A rollout containing both tags in the correct order receives 1.0. A typo, reversed order, or missing tag receives 0.0. During training, excessive format credit can keep average reward high while evaluation accuracy and response length decline; lowering the weight or conditioning the reward on correctness can mitigate this failure mode. section 7.6.1 section 7.6.2

Figure 7.17 Implementing a format reward that encourages the model to generate <think>...</think> tokens

Use the workflow figure to inspect how the format reward supplements correctness reward and encourages <think>...</think> output structure. section 7.6

How Build a Reasoning Model (From Scratch) covers it

How deep it goes

Implements ordered tag validation, loss integration, prompt changes, and reward-balance diagnostics, but defers broader GRPO extensions.

Examples

Uses <think> tags, tokenizer encodings, malformed-tag tests, updated prompts, and GRPO training metrics.

What it assumes

Assumes an existing GRPO/RLVR loop and teaches tokenizer support, tag validation, reward weighting, and prompt rendering.

What you can do after

Apply it via a GRPO loss function to reward correctly ordered reasoning tags and monitor whether formatting overwhelms answer correctness.

How it connects

Links GRPO, reinforcement learning with verifiable rewards, correctness rewards, tokenizers, and KL regularization.

liveBookmanning.com

Read Build a Reasoning Model (From Scratch) on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Reinforcement Learning from Human FeedbackA format reward is a reward signal assigned according to whether a model output follows a required structural presentation convention.

Compare how 2 books explain response-formatting reward →

Related concepts

• <think> tagaddresses · A think tag is a markup element that delimits reasoning-style text from a model's final answer text.
• accuracycontrasts with · Accuracy is the degree to which an output, prediction, or measurement agrees with a correct or reference outcome under a specified criterion.
• reward signalproduces · A reward signal is an evaluative feedback value representing the desirability or payoff of an action, outcome, or state.
• reinforcement learning with verifiable rewardsused by · Reinforcement learning with verifiable rewards is a training approach that optimizes a model using rewards derived from objectively checkable outcomes.
• verifiable rewardcontrasts with · A verifiable reward is a reward signal whose correctness can be determined by an objective criterion or procedure.

This explainer was generated by Manning's knowledge graph from the text of Build a Reasoning Model (From Scratch). The section links point to the parts of the book it draws on.