liveBook · Manning

Evaluation dataset

As explained in Designing AI Systems

An evaluation dataset is a collection of test cases containing inputs, expected outputs, and metadata for assessing an AI system or change.

An evaluation dataset is a reusable benchmark of test cases for measuring AI-system quality. Run prompt alternatives against the same cases to replace intuition with comparable results before deployment. section 2.1.6 section 8.3.1

A test case can include:

  • An input query, an ideal response, key elements for automated scoring, and tags for filtering. section 8.3.1
  • Hand-curated cases for known scenarios, supplemented with production traces that reveal difficult or safety-sensitive edge cases. section 8.3.1
  • Human review before production traces become test cases, to avoid adding low-quality or misleading examples. section 8.3.1

Patient intake benchmark

A case asking what documents to bring can specify an ideal answer and key elements such as an insurance card, photo ID, and medical records. Reuse it to compare prompt versions. section 2.1.6

Listing 8.4 Building evaluation datasets from curated cases and production traces
# Start with hand-curated test cases
benchmark = platform.experiments.create_dataset(
    name="intake-quality-benchmark",
    test_cases=[
        {"input_query": "What documents do I need for my "
                       "first appointment?",
         "ideal_response": "Please bring your insurance "
                         "card, photo ID, and medical records.",
         "key_elements": ["insurance card", "photo ID",
                        "medical records"],
         "tags": ["documents", "first-visit"]},
        # ... more curated cases
    ],
)
 
# Augment with real production edge cases
low_quality = platform.observability.query_traces(    #A
    workflow_id="patient-intake",
    start_time=last_week,
    max_score={"helpfulness": 0.5},    #B
    limit=50,
)
 
edge_cases = platform.observability.query_traces(
    workflow_id="patient-intake",
    tags=["guardrail_triggered"],    #C
    limit=50,
)
 
platform.experiments.add_from_production(    #D
    dataset_name="intake-quality-benchmark",
    trace_ids=[t.trace_id for t in low_quality + edge_cases],
    require_human_review=True,
)

Shows a dataset built from curated cases and augmented with production traces selected for low helpfulness or guardrail triggers. section 8.3.1

Refresh the dataset with reviewed production samples as user behavior changes; a static benchmark can miss emerging cases and regressions. section 8.3.1

How Designing AI Systems covers it

How deep it goes
Defines test-case fields and describes curated-case creation, production-trace selection, and human review.
Examples
Uses a patient document-request case and production traces with low helpfulness scores or guardrail triggers.
What it assumes
Assumes access to production traces and quality signals for finding candidate edge cases; also supports hand-curated cases.
What you can do after
Evaluate prompt versions against the same test cases and augment the benchmark with reviewed production examples.
How it connects
Connects datasets to production traces, quality scores, observability, offline evaluation, A/B testing, and human annotation.

Also explained in

Compare how 41 books explain evaluation dataset →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Designing AI Systems. The section links point to the parts of the book it draws on.