Experimentation routes live traffic among alternatives and measures outcomes. Unlike offline evaluation, it exposes prompt, model, or retrieval changes to real user behavior, where benchmark performance may not predict production results. section 8.4
A production experiment defines:
- Two or more variants, such as different prompts, models, or retrieval configurations. section 8.4
- Traffic allocation describing what percentage of requests each variant receives. section 8.4
- Success metrics, including whether higher or lower values are preferable. section 8.4
- Stable assignment using consistent hashing so one patient remains on the same variant throughout an experiment. section 8.4
Non-invasive workflow integration
A workflow calls assign_variant at the beginning, conditionally selects the assigned prompt version, invokes the existing Model Service, and records the outcome afterward. With no active experiment, assignment returns None and the default prompt remains active. section 8.4
Figure 8.4 A/B testing architecture. Incoming requests are assigned to variants through consistent hashing on the patient ID, ensuring the same patient always sees the same variant (preventing jarring style changes mid-conversation). The 50/50 traffic split sends half to the current prompt and half to the new candidate. Outcome metrics (quality scores, turns to resolution, cost) are collected for both variants and compared with statistical significance testing. The experiment concludes when enough samples exist to confidently recommend a winner.

The architecture diagram shows a 50/50 split, consistent patient assignment, metric collection, statistical comparison, and a sample-based stopping point for recommending a winner. section 8.4



















