Code-based evaluation automatically grades an agent’s response with predefined rules or executable programs. It provides fast, objective, reproducible checks as agents, prompts, models, and data change. section 10.3
Common evaluator strategies include:
- Exact string matching for answers with a fixed expected result. section 10.3
- Strict format validation, including JSON schemas and required fields. section 10.3
- Regular-expression or parse-tree comparisons for structured outputs. section 10.3
- Execution-based evaluation that runs generated code or SQL against test cases. section 10.3
- Constraint checks for prohibited words, length limits, and policy violations. section 10.3
Regression checking
A CI pipeline can rerun deterministic format, correctness, execution, and constraint checks after a prompt or model change, exposing regressions without manual grading. section 10.3
