An A/B test is a controlled production experiment that routes users or traffic to alternative variants and compares measured outcomes. In AI systems, variants may change prompts, models, retrieval configurations, or combinations of them. Unlike offline evaluation, it exposes alternatives to real user behavior and validates whether a proposed improvement survives production conditions. section 8.4 section 8.0
A production-ready experiment defines:
- Variants, such as a current prompt and a candidate prompt, model configuration, or retrieval configuration. section 8.4
- Traffic allocation, specifying what percentage of requests each variant receives. section 8.4
- Success metrics, including quality, resolution time, cost, latency, or task-specific accuracy, with the desired direction for each metric. section 8.4 section 10.11
- A statistically controlled stopping decision based on sample size and significance, rather than an early impression. section 8.1 section 8.6
Consistent assignment keeps an identity tied to one variant throughout an experiment. Consistent hashing combines the experiment name with a patient or user ID to produce a deterministic bucket, preventing style or behavior changes between interactions. Experiment-aware workflows remain unchanged when no experiment is active: assignment returns no variant, the default configuration runs, and outcome recording becomes a no-op. section 8.4 section 8.6
Testing a prompt change
A deployed assistant compares its current prompt with a tools-first candidate using a 50/50 split. It measures response quality, tool-selection accuracy, cost, and latency, requires at least 500 samples, and promotes the candidate only when the evidence supports improvement without unacceptable regressions. section 10.11
Figure 8.4 A/B testing architecture. Incoming requests are assigned to variants through consistent hashing on the patient ID, ensuring the same patient always sees the same variant (preventing jarring style changes mid-conversation). The 50/50 traffic split sends half to the current prompt and half to the new candidate. Outcome metrics (quality scores, turns to resolution, cost) are collected for both variants and compared with statistical significance testing. The experiment concludes when enough samples exist to confidently recommend a winner.

Use the architecture diagram to trace deterministic assignment, traffic splitting, outcome collection, significance testing, and experiment completion. section 8.4




