Evaluation systematically tests AI changes against evidence and criteria, helping determine whether a prompt, model configuration, or retrieval parameter should move from draft to production. section 8.3
Use three complementary evaluation modes:
- Offline evaluation tests changes against curated datasets to catch regressions before deployment. section 8.3
- Online evaluation examines live production traffic to catch problems curated datasets miss. section 8.3
- Human evaluation uses annotation queues to provide ground truth for the other modes. section 8.3
Listing 2.8 Systematic prompt optimization through Experimentation Service
evaluation_dataset = [ #A
{ #A
"input_query": "What documents do I need for my first appointment?", #A
"ideal_response": "Please bring your insurance card, a valid photo ID, and any medical records from previous doctors.", #A
"key_elements": ["insurance card", "photo ID", "medical records"] #A
}, #A
… #A
] #A
for prompt_name in ["professional_formal", "warm_conversational"]: #B
platform.experiments.register_target(name=prompt_name, version=1) #B
platform.experiments.create_dataset( #C
name="patient_intake_eval", #C
test_cases=evaluation_dataset) #C
for progress in platform.experiments.run_evaluation( #D
dataset_name="patient_intake_eval", #D
targets=["professional_formal", "warm_conversational"], #D
metrics=["content_accuracy", "safety_compliance", "helpfulness"]): #D
results = progress.results #DCompare two patient-intake prompt variants against the same dataset of questions, ideal responses, and key elements. section 2.1.6




























