An evaluation dataset is a reusable benchmark of test cases for measuring AI-system quality. Run prompt alternatives against the same cases to replace intuition with comparable results before deployment. section 2.1.6 section 8.3.1
A test case can include:
- An input query, an ideal response, key elements for automated scoring, and tags for filtering. section 8.3.1
- Hand-curated cases for known scenarios, supplemented with production traces that reveal difficult or safety-sensitive edge cases. section 8.3.1
- Human review before production traces become test cases, to avoid adding low-quality or misleading examples. section 8.3.1
Patient intake benchmark
A case asking what documents to bring can specify an ideal answer and key elements such as an insurance card, photo ID, and medical records. Reuse it to compare prompt versions. section 2.1.6
Listing 8.4 Building evaluation datasets from curated cases and production traces
# Start with hand-curated test cases
benchmark = platform.experiments.create_dataset(
name="intake-quality-benchmark",
test_cases=[
{"input_query": "What documents do I need for my "
"first appointment?",
"ideal_response": "Please bring your insurance "
"card, photo ID, and medical records.",
"key_elements": ["insurance card", "photo ID",
"medical records"],
"tags": ["documents", "first-visit"]},
# ... more curated cases
],
)
# Augment with real production edge cases
low_quality = platform.observability.query_traces( #A
workflow_id="patient-intake",
start_time=last_week,
max_score={"helpfulness": 0.5}, #B
limit=50,
)
edge_cases = platform.observability.query_traces(
workflow_id="patient-intake",
tags=["guardrail_triggered"], #C
limit=50,
)
platform.experiments.add_from_production( #D
dataset_name="intake-quality-benchmark",
trace_ids=[t.trace_id for t in low_quality + edge_cases],
require_human_review=True,
)Shows a dataset built from curated cases and augmented with production traces selected for low helpfulness or guardrail triggers. section 8.3.1
Refresh the dataset with reviewed production samples as user behavior changes; a static benchmark can miss emerging cases and regressions. section 8.3.1








































