liveBook · Manning

A/B test

As explained in Designing AI Systems

A controlled experiment that compares alternative variants by assigning them to different user or traffic groups and measuring outcomes.

An A/B test is a controlled production experiment that routes users or traffic to alternative variants and compares measured outcomes. In AI systems, variants may change prompts, models, retrieval configurations, or combinations of them. Unlike offline evaluation, it exposes alternatives to real user behavior and validates whether a proposed improvement survives production conditions. section 8.4 section 8.0

A production-ready experiment defines:

  • Variants, such as a current prompt and a candidate prompt, model configuration, or retrieval configuration. section 8.4
  • Traffic allocation, specifying what percentage of requests each variant receives. section 8.4
  • Success metrics, including quality, resolution time, cost, latency, or task-specific accuracy, with the desired direction for each metric. section 8.4 section 10.11
  • A statistically controlled stopping decision based on sample size and significance, rather than an early impression. section 8.1 section 8.6

Consistent assignment keeps an identity tied to one variant throughout an experiment. Consistent hashing combines the experiment name with a patient or user ID to produce a deterministic bucket, preventing style or behavior changes between interactions. Experiment-aware workflows remain unchanged when no experiment is active: assignment returns no variant, the default configuration runs, and outcome recording becomes a no-op. section 8.4 section 8.6

Testing a prompt change

A deployed assistant compares its current prompt with a tools-first candidate using a 50/50 split. It measures response quality, tool-selection accuracy, cost, and latency, requires at least 500 samples, and promotes the candidate only when the evidence supports improvement without unacceptable regressions. section 10.11

Figure 8.4 A/B testing architecture. Incoming requests are assigned to variants through consistent hashing on the patient ID, ensuring the same patient always sees the same variant (preventing jarring style changes mid-conversation). The 50/50 traffic split sends half to the current prompt and half to the new candidate. Outcome metrics (quality scores, turns to resolution, cost) are collected for both variants and compared with statistical significance testing. The experiment concludes when enough samples exist to confidently recommend a winner.

Use the architecture diagram to trace deterministic assignment, traffic splitting, outcome collection, significance testing, and experiment completion. section 8.4

How Designing AI Systems covers it

How deep it goes
Explains variants, allocation, assignment, metrics, and significance, but does not detail statistical test formulas.
Examples
Uses prompt comparisons measuring response quality, tool-selection accuracy, cost, latency, and sample size.
What it assumes
Assumes production traffic, versioned artifacts, measurable outcomes, and observability traces connected to evaluation workflows.
What you can do after
Operate the Experimentation Service to assign variants, record outcomes, compare metrics, and promote or reject production changes.
How it connects
Connects observability, offline evaluation, online evaluation, evaluation datasets, consistent hashing, and statistical significance.

Also explained in

Compare how 5 books explain A/B test →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Designing AI Systems. The section links point to the parts of the book it draws on.