liveBook · Manning

Reward function

As explained in LLM Customization and Fine-Tuning

A function that assigns a numerical score to an output, action, or sequence of actions to represent its quality.

A reward function converts a model response into a numerical score. In reinforcement fine-tuning, it grades sampled outputs so the policy can be updated to favor higher-scoring responses. When the answer is verifiable, a programmatic grader can replace human preference annotations and reward-model training. section 8.5.2 section 8.5

Match the grader to what can be checked:

  • For structured extraction, parse the response and compare required fields with known values; return zero if parsing fails. section 8.5.2
  • For math or code, compare the final answer with ground truth or run generated code against test cases. section 8.5.2
  • For multi-step tasks, score the whole trajectory when the workflow and its evaluation criteria can be represented programmatically. section 8.5.1

Invoice extraction

A verifier checks an extracted invoice number, amount, and due date against expected values. It returns the fraction of matching fields, or zero for an unparseable response: a fully correct response scores 1.00, one wrong field scores 0.67, and prose that is not JSON scores 0.00. section 8.5.2

Listing 8.5 verifiable_reward.py (key sections)
# (key sections -- full source in code/chapter08/scripts/verifiable_reward.py)
REQUIRED_FIELDS = ("invoice_number", "amount", "due_date")
 
def invoice_reward(response: str, expected: dict) -> float: #A
    if parsed is None:                                      #B
        return 0.0
    hits = sum(1 for f in REQUIRED_FIELDS
               if str(parsed.get(f, "")).strip()
               == str(expected[f]).strip())  #C
    return hits / len(REQUIRED_FIELDS)

Inspect the Python-style invoice verifier, which illustrates parsing, field-by-field comparison, and a numeric score. section 8.5.2

A useful reward is deterministic, inexpensive to evaluate, and aligned with the quality dimension that matters. GRPO requires a callable function that scores individual responses; its compute cost rises with the number of generated responses, making cheap checks such as string matching more practical than judge-model scoring. DPO, by contrast, does not require a separate reward model or RL loop. section 8.5.1 section 8.3.3 section 8.2

How LLM Customization and Fine-Tuning covers it

How deep it goes
Defines a callable output-to-score function and traces scoring into policy updates, but limits implementation detail to an invoice example.
Examples
Uses invoice-field matching, math answers, code tests, and diagnostic conversations as scoring cases.
What it assumes
Requires known correct answers or expert-designed criteria that can be scored programmatically.
What you can do after
Build a callable verifier that returns numeric response scores, then use it in a GRPO training setup.
How it connects
Connects verifiable rewards to RFT and GRPO, while contrasting DPO's preference-based approach and lack of a separate reward model.

Also explained in

Compare how 3 books explain reward function →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.