A judge program is a DSPy program that scores another program’s output. For dialog summarization, it can assess a generated summary inside a metric function. A simple judge compares the output with a ground-truth summary; a question-based judge instead examines the dialog and generated summary, so it can evaluate summaries for dialogs without reference summaries. section 8.6.2 section 8.6.10
Build and check the judge separately from the summarizer:
- Define its inputs and criteria. A Q&A signature takes a dialog and summary and returns scores from 0.0 to 1.0 for salience and named entities. The example treats their average above 0.5 as a positive judgment. section 8.6.10
- Prepare labeled judge examples containing both good and bad summaries. The example uses ground-truth summaries as good cases and summaries borrowed from other dialogs as bad cases; testing only good cases could reward a judge that always approves. section 8.6.5
- Evaluate the judge against known labels, optimize it if needed, and check it again on a hold-out set. Keep judge and summarizer data separate to avoid leakage. section 8.6.10 section 8.6.2
Listing 8.22 Q&A signature and program
class QASignature(dspy.Signature):
'''Your task is to judge the summary of a dialog'''
dialog: str = dspy.InputField()
summary: str = dspy.InputField()
salient: float = dspy.OutputField(desc="On a scale from 0.0 to 1.0")
named_entities: float = dspy.OutputField(desc="On a scale from 0.0 to 1.0")
def get_final_score(prediction: dspy.Prediction): #1
return ((prediction.salient + prediction.named_entities) / 2) > 0.5
qa_judge = dspy.ChainOfThought(QASignature)
The signature makes each scoring criterion an explicit output field; ChainOfThought turns it into a callable judge program. section 8.6.10
Once checked, the judge can become the summarizer’s metric: pass it the original dialog and the summarizer’s prediction, then return its decision. Judge quality matters because errors in that decision affect how summarizer outputs are evaluated. Criteria that need no language-model judgment, such as raw word count, can instead be measured directly in the metric function. section 8.6.10 section 8.6.2
Listing 8.27 Metric function for evaluating summaries using a Q&A criterion
def qa_metric(example: dspy.Example,
prediction: dspy.Prediction, trace=None):
judge_pred = simba_qa_judge(dialog=example['dialog'],
summary=prediction['summary'])
return get_final_score(judge_pred) #1
The metric function shows the optimized judge being called on a dialog and generated summary, with its decision returned as the metric. section 8.6.10
