DSPy’s `Evaluate` runs a program over a development set with a metric, replacing a hand-written per-example evaluation loop. Configure a thread count and display options to evaluate programs more consistently and inspect results. section 4.3.2
Listing 4.13 Evaluating the baseline with DSPy Evaluate
import dspy
import time
import os
os.environ['OPENAI_API_KEY'] = OPENAI_API_KEY #1
def evaluate_baseline(model="openai/gpt-4o-mini", num_threads=10):
lm = dspy.LM(model, cache=False)
dspy.configure(lm=lm)
classifier = dspy.Predict(IntentSignature) #2
evaluator = dspy.Evaluate(devset=dev_set, #3
num_threads=num_threads,
display_progress=True,
display_table=5,
provide_traceback=True,
max_errors=100000)
start_time = time.time()
results = evaluator(classifier, metric=validate_answer) #4
end_time = time.time()
print(f"total time: {end_time - start_time}")
return results.score
overall_score = evaluate_baseline(num_threads=10)
See a runnable evaluation setup that configures a language model, program, development set, metric, and evaluation options. section 4.3.2
