liveBook · Manning

Format bonus

As explained in Applied Reinforcement Learning

An additional reward for presenting an answer in a required format, independent of whether its content is correct.

A format bonus rewards a required answer format separately from answer correctness. In a FinQA verifier, a response containing a boxed answer earns 0.1, whether or not the number is correct; the numerical check independently adds 1.0 when correct. section 11.2

The two checks combine into four outcomes:

  • Correct and boxed: 1.1; correct but unboxed: 1.0. section 11.2
  • Wrong but boxed: 0.1; wrong and unboxed: 0.0. section 11.2
Figure 11.2 The RLVR reward signal structure.

The diagram summarizes how format and correctness rewards add together. section 11.2

This small extra reward gives learning a graded signal: in a GRPO example, boxed-but-wrong answers receive a less severe penalty than unboxed-and-wrong answers. Across training, the format rate rises from 89.8% to 99.1%, illustrating how a cheap, clearly rewarded behavior can become more reliable. section 11.4 section 11.7.6

How Applied Reinforcement Learning covers it

How deep it goes
Defines a 0.1 boxed-answer reward, combines it with correctness, and traces its effect through GRPO advantages.
Examples
Compares four boxed-or-unboxed correctness outcomes and an eight-response GRPO group.
What it assumes
Assumes a verifier can extract a boxed number and compare it with a gold answer.
What you can do after
Evaluate/understand how a format bonus distinguishes correct answers from formatting failures and shapes relative rewards.
How it connects
Connects answer-format rewards to numerical verification and group-relative policy optimization (GRPO).

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.