Consistency measures how similar a language model’s responses are across repeated calls with the same input and configuration. DSPy’s default cache can hide variation by returning the first response again; disable caching when testing consistency. section 4.6 section 2.4.4
Two repeated-test approaches answer different questions:
- Repeat each test example and combine the scores to assess accuracy across calls. This gives a broader accuracy assessment, but does not isolate consistency. section 4.6
- For a consistency-focused measure, count the predictions for each example. One simple measure is the number that differ from the most frequent prediction; it does not account for accuracy. section 4.6
Listing 4.18 Testing for consistency
from collections import Counter
import dspy
lm = dspy.LM('gpt-4o-mini', cache=False)
dspy.configure(lm=lm)
classifier = dspy.Predict(IntentSignature)
for example in dev_set:
examples = [example] * 10
evaluator = dspy.Evaluate(devset=examples, num_threads=10,
display_progress=False,
display_table=False,provide_traceback=False,
max_errors=100000)
res = evaluator(classifier, metric=validate_answer)
predictions = [prediction.intent_label for _, prediction, _ in res.results]
print(predictions)
print(Counter(predictions))
A DSPy example repeats each development-set item ten times, evaluates the calls, and counts the resulting intent labels. section 4.6
Repeated evaluation can extract more value from a small test set and help distinguish prompts with similar scores. It also takes more time and costs more, so reserve it for cases where a more accurate assessment is needed. section 4.6
