A verification pipeline evaluates free-form model answers by loading a language model, processing responses, and scoring correctness against a dataset. Its math-verification workflow extracts boxed answers, normalizes mathematical notation, and checks symbolic equivalence with SymPy. section 3.10
The pipeline proceeds through these implementation stages:
- Load a saved model checkpoint or download a pretrained checkpoint for evaluation. section 6.12
- Generate answers and extract boxed content with string parsing, using fallbacks when boxes are missing. section 3.10
- Normalize diverse answers by stripping LaTeX and converting mathematical notation. section 3.10
- Compare expressions symbolically with SymPy and aggregate accuracy on MATH-500. section 3.10
Evaluating a saved checkpoint
Run the evaluation utility with a checkpoint path and a 500-example dataset, then compare accuracy and average response length across models. section 6.12
Figure 6.21 We’ll now load the saved model checkpoints and evaluate them.
Use the workflow figure to orient checkpoint loading and evaluation as connected stages. section 6.12
