Early stopping ends an evaluation early when a candidate prompt appears unlikely to perform well enough. Screening candidates on a subset of validation examples can save evaluation time and cost. section 5.2.2 section 4.3.2
A straightforward screening rule evaluates a candidate on an initial subset, then:
- Stops if the score falls below a fixed threshold; one example checks 50 of 300 validation examples and stops below 80%. section 5.2.2
- Continues to the remaining examples if the candidate passes. A weak result on the first subset may wrongly discard a strong prompt. section 5.2.2
- Can use several checkpoints or compare a candidate’s score with previously tested prompts to prune candidates unlikely to rank well. section 5.2.2
Manual evaluation code can add early stopping when a candidate performs poorly on an initial share of test cases. A standard evaluation framework offers a cleaner evaluation workflow, while manual code allows this custom exit condition. section 4.3.2
Listing 5.4 Calling the LabeledFewShot optimizer repeatedly in a loop
import dspy
from dspy.teleprompt import LabeledFewShot
best_score = -1
best_model = None
for num_examples in range(1, 21): #1
for _ in range(10): #2
labeled_few_shot_optimizer = LabeledFewShot(k=num_examples)
few_shot_model =
labeled_few_shot_optimizer.compile(student=intent_classifier,
trainset=train_set)
evaluator= dspy.Evaluate(devset=val_set[:50], #3
num_threads=5,
display_progress=False,
display_table=False,
provide_traceback=False,
max_errors=0)
results_part1 = evaluator(few_shot_model, metric=validate_answer)
if results_part1.score < 80.0: #4
continue
evaluator= dspy.Evaluate(devset=val_set[50:], #5
num_threads=5,
display_progress=False,
display_table=False,
provide_traceback=False,
max_errors=0)
results_part2 = evaluator(few_shot_model, metric=validate_answer)
results = (results_part1.score + results_part2.score * 5) / 6 #6
if results > best_score:
best_score = results
best_model = few_shot_model
print(best_score)
See the Python loop that evaluates candidate prompts on an initial subset and applies a score cutoff. section 5.2.2
