liveBook · Manning

Answer checking pipeline

As explained in Build a Reasoning Model (From Scratch)

A deterministic workflow that extracts a model's answer and compares it to the reference for correctness.

An answer-checking pipeline turns generated reasoning into a measurable prediction: render a prompt, generate text, extract the final candidate, compare it with the reference, and aggregate correctness across MATH-500. section 3.9

A failed mini-evaluation

For “Compute 1/2 + 1/6,” the pipeline extracts 1/3, compares it with the ground truth 2/3, and returns False. Prompt choices can affect accuracy: a concise boxed response may be wrong even when a longer response succeeds. section 3.9

How Build a Reasoning Model (From Scratch) covers it

How deep it goes
Explains implementation from prompt rendering through dataset aggregation, while deferring broader inference improvements to later work.
Examples
Uses boxed-answer prompts, a fraction-addition mini-demo, MATH-500 subsets, device comparisons, progress output, and JSONL results.
What it assumes
Assumes a loaded model, tokenizer, device, and dataset while teaching the orchestration of generation, extraction, grading, and reporting.
What you can do after
Apply via render_prompt, generate_text_stream_concat, extract_final_candidate, and grade_answer to compute correctness and accuracy.
How it connects
Connects prompt templates, text generation, answer extraction, model evaluation, and the MATH-500 dataset.
Cover of Build a Reasoning Model (From Scratch)
liveBookmanning.com
Read Build a Reasoning Model (From Scratch) on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a Reasoning Model (From Scratch). The section links point to the parts of the book it draws on.