An objective function is a quantitative indicator that aggregates evaluation judgments into a value trackable across cases or over time. In agent evaluation, it turns quality criteria into measurable signals for dashboards, alerts, and performance tracking. section 10.1.1 section 10.2.4
Build the indicator from explicit evaluation criteria rather than vague impressions. A rubric defines how each case is judged; the objective function aggregates those judgments into numbers that reveal regressions and guide improvement. section 10.2 section 10.5
A practical evaluation objective can track:
- Overall pass rate: the fraction of cases passing answer relevancy, citation reliability, and requirement compliance. section 10.2.4
- Component rates: separate pass rates for answer relevancy, citation reliability, and requirement compliance. section 10.2.4
- Operational signals: latency, task-completion time, tool-call frequency, token use, cost, context-window use, and LLM-call errors. section 10.1.1
From rubric to trackable value
For each evaluated response, record PASS or FAIL for answer relevancy, citation reliability, and requirement compliance. Compute overall pass rate as all-pass cases divided by total evaluations, then monitor the result over time. section 10.2.4
Figure 10.2 Workflow for defining agent evaluation criteria, from open coding to defining metrics
Use the workflow diagram to trace how error analysis becomes a failure taxonomy, rubric, and set of trackable metrics. section 10.2
Treat the objective function as part of a continuous evaluation loop: execution data supports error analysis, taxonomies refine rubrics and metrics, and updated metrics guide test-set and deployment improvements. section 10.5
