An evaluator uses a language model to judge another model’s predictions. In DSPy, the judge can be called from a metric function and improved as a program through prompt optimization. section 8.6.2
Plan for the evaluator’s role and cost:
- Use the judge during optimization and evaluation, not production; repeated candidate predictions can generate many judge calls. section 8.6.3
- Select an appropriate judge model and optimize its instructions or demonstrations. Keep judge and summarizer data separate to avoid leakage. section 8.6.2
- Early stopping or minibatches can reduce calls, but a capable judge may be expensive. section 8.6.3
Listing 8.14 DSPy metric function using LLM-as-a-judge
def evaluate_summary(example: dspy.Example, prediction: dspy.Prediction, trace=None):
judge = dspy.Predict("ground_truth_summary, system_summary ->
summary_quality: float")
return judge(
ground_truth_summary=example.summary,
system_summary=prediction.summary)
evaluate_summary(summarizer_train_data[0], pred)
See a DSPy metric function that asks a judge to score a predicted summary against a ground-truth summary. section 8.6.2
