SymPy provides the symbolic parsing and equivalence-checking step in a math verifier. It converts extracted, normalized model answers into comparable expressions instead of relying on fragile string equality. section 3.10 section 3.6
A robust SymPy-based check follows this workflow:
- Reject missing or excessively long expressions, then parse with standard transformations and implicit multiplication support. section 3.6
- Simplify basic arithmetic during parsing and return a null result for malformed model output or parser errors. section 3.6
- Try exact string equality first; otherwise parse both expressions and test whether simplifying their difference produces zero. section 3.6
Equivalent fractions
SymPy treats 28/6 and 14/3 as the same mathematical value, and can likewise accept 0.5 and 1/2 when checking a model prediction against ground truth. section 3.6
Figure 3.8 An illustration of how an LLM-generated answer is checked against the correct reference answer (ground truth). The LLM’s final boxed answer is extracted, normalized, and compared to the correct answer provided in the dataset. If they match, the response is graded as correct.
Follow the verification flow from boxed-answer extraction and normalization to symbolic comparison with the dataset ground truth. section 3.6
