liveBook · Manning

Experimentation

As explained in Designing AI Systems

An experiment is a controlled procedure that assigns alternatives to traffic and measures their outcomes.

Experimentation routes live traffic among alternatives and measures outcomes. Unlike offline evaluation, it exposes prompt, model, or retrieval changes to real user behavior, where benchmark performance may not predict production results. section 8.4

A production experiment defines:

  • Two or more variants, such as different prompts, models, or retrieval configurations. section 8.4
  • Traffic allocation describing what percentage of requests each variant receives. section 8.4
  • Success metrics, including whether higher or lower values are preferable. section 8.4
  • Stable assignment using consistent hashing so one patient remains on the same variant throughout an experiment. section 8.4

Non-invasive workflow integration

A workflow calls assign_variant at the beginning, conditionally selects the assigned prompt version, invokes the existing Model Service, and records the outcome afterward. With no active experiment, assignment returns None and the default prompt remains active. section 8.4

Figure 8.4 A/B testing architecture. Incoming requests are assigned to variants through consistent hashing on the patient ID, ensuring the same patient always sees the same variant (preventing jarring style changes mid-conversation). The 50/50 traffic split sends half to the current prompt and half to the new candidate. Outcome metrics (quality scores, turns to resolution, cost) are collected for both variants and compared with statistical significance testing. The experiment concludes when enough samples exist to confidently recommend a winner.

The architecture diagram shows a 50/50 split, consistent patient assignment, metric collection, statistical comparison, and a sample-based stopping point for recommending a winner. section 8.4

How Designing AI Systems covers it

How deep it goes
Defines variants, allocation, metrics, hashing, workflow hooks, and statistical comparison, but does not detail statistical methods.
Examples
Uses patient-intake prompt variants, 50/50 traffic, quality scores, turns to resolution, and cost as comparison artifacts.
What it assumes
Assumes familiarity with live traffic, workflows, model services, prompt versions, and offline evaluation.
What you can do after
Operate production comparisons that assign stable variants, collect outcomes, and recommend a winner after sufficient samples.
How it connects
Connects offline evaluation, prompt versions, traffic allocation, consistent hashing, success metrics, and statistical significance testing.

Also explained in

Compare how 20 books explain experimentation →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Designing AI Systems. The section links point to the parts of the book it draws on.