Completeness measures summary recall: how well the summary covers the source’s important ideas. Unlike groundedness, it measures coverage rather than whether stated ideas appear in the source. It can be judged against the source or a reference summary; groundedness requires the source. section 8.6.1
Listing 8.18 Completeness signature and program
from dspy import Signature, InputField, OutputField
class SummaryCompleteness(Signature):
"""
Estimate the completeness of a system's responses, against the ground
truth summary.
You will first enumerate key ideas in each of the inputs, discuss their
overlap, and then report completeness.
"""
dialog: str = InputField()
summary: str = InputField()
dialog_key_ideas: str = OutputField(desc=
"enumeration of key ideas in the dialog")
system_summary_response_key_ideas: str = OutputField(desc=
"enumeration of key ideas in the system summary response")
discussion: str = OutputField(desc=
"discussion of the overlap between ground truth and system response")
completeness: float = OutputField(desc=
"fraction (out of 1.0) of ground truth covered by the system response
summary")
completeness_predictor = dspy.Predict(SummaryCompleteness)
pred = completeness_predictor(
dialog=judge_val_data[0].dialog,
summary=judge_val_data[0].summary)
print(pred)
A DSPy signature takes a dialog and summary, asks a judge to enumerate key ideas and discuss their overlap, then returns a completeness score. section 8.6.7
Interpreting a score
A dialogue summary scores 0.8 when it captures most topics but omits details such as leadership roles and the specific sports discussed. Test judge quality on a full validation set; one example is not enough. section 8.6.7
Completeness works alongside groundedness: a summary can cover important ideas yet include unsupported claims, or stay grounded while omitting key points. Combine the scores with a simple average or an F1-based DSPy module; judge programs can also be optimized. section 8.6.1 section 8.6.8
