liveBook · Manning

Evaluation

As explained in Designing AI Systems

Evaluation is the systematic assessment of an AI system or change against defined evidence and criteria.

Evaluation systematically tests AI changes against evidence and criteria, helping determine whether a prompt, model configuration, or retrieval parameter should move from draft to production. section 8.3

Use three complementary evaluation modes:

  • Offline evaluation tests changes against curated datasets to catch regressions before deployment. section 8.3
  • Online evaluation examines live production traffic to catch problems curated datasets miss. section 8.3
  • Human evaluation uses annotation queues to provide ground truth for the other modes. section 8.3
Listing 2.8 Systematic prompt optimization through Experimentation Service
evaluation_dataset = [    #A
    {    #A
        "input_query": "What documents do I need for my first appointment?",    #A
        "ideal_response": "Please bring your insurance card, a valid photo ID, and any medical records from previous doctors.",    #A
        "key_elements": ["insurance card", "photo ID", "medical records"]    #A
    },    #A
    …    #A
]    #A
 
for prompt_name in ["professional_formal", "warm_conversational"]:    #B
    platform.experiments.register_target(name=prompt_name, version=1)    #B
 
platform.experiments.create_dataset(    #C
    name="patient_intake_eval",    #C
    test_cases=evaluation_dataset)    #C
 
for progress in platform.experiments.run_evaluation(    #D
        dataset_name="patient_intake_eval",    #D
        targets=["professional_formal", "warm_conversational"],    #D
        metrics=["content_accuracy", "safety_compliance", "helpfulness"]):    #D
    results = progress.results    #D

Compare two patient-intake prompt variants against the same dataset of questions, ideal responses, and key elements. section 2.1.6

How Designing AI Systems covers it

How deep it goes
Defines three evaluation modes and their roles, then illustrates dataset-based prompt comparison without detailing scoring algorithms.
Examples
Compares formal and conversational patient-intake prompts using questions, ideal responses, and key elements as test data.
What you can do after
Evaluate prompt or configuration changes against shared benchmarks and use results to guide improvement or promotion.
How it connects
Links evaluation to prompt versions, model configurations, retrieval parameters, curated datasets, production traffic, and human annotation.

Also explained in

Compare how 29 books explain evaluation →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Designing AI Systems. The section links point to the parts of the book it draws on.