liveBook · Manning

Rule-based scoring function

As explained in Build a Reasoning Model (From Scratch)

A manually specified scoring method that assigns numeric values to model outputs using explicit rules rather than learned parameters.

A rule-based scoring function ranks generated responses with explicit heuristics instead of learned parameters. It supports inference-time self-refinement when correctness cannot be directly verified, prioritizing format and brevity. section 5.3

A practical scorer can combine:

  • A boxed-answer bonus for cleanly extracted final candidates. section 5.3
  • A smaller number-extraction bonus when no boxed answer is available. section 5.3
  • An exponential brevity contribution that helps break ties without dominating extraction quality. section 5.3

Heuristic scorer

Extract a boxed candidate, then a number-only candidate, then a full-text fallback; add the corresponding bonus and 1.5 × exp(−length / brevity_bonus).

How Build a Reasoning Model (From Scratch) covers it

How deep it goes
Defines fallback extraction, weighted bonuses, and an exponential brevity term, but stops short of learned or probability-based scoring.
Examples
Uses boxed answers, number extraction, full-text fallback, response pairs, and plotted brevity curves.
What it assumes
Assumes generated responses and candidate-extraction utilities, while making heuristic scoring explicit from a simple baseline.
What you can do after
Build it from scratch by combining extraction rules, weighted bonuses, and length-based scoring.
How it connects
Connects response ranking with self-refinement, LLM-as-a-judge, likelihood scores, and verification.
Cover of Build a Reasoning Model (From Scratch)
liveBookmanning.com
Read Build a Reasoning Model (From Scratch) on liveBook Get the book at manning.com

This explainer was generated by Manning's knowledge graph from the text of Build a Reasoning Model (From Scratch). The section links point to the parts of the book it draws on.