An experimentation-as-a-service platform provides shared infrastructure for testing and improving AI applications. It replaces blind changes with versioned targets, quality measurement, controlled comparisons, and a path from evaluation to production. section 1.1.3 section 8.0
The service complements observability rather than replacing it. Observability records traces, costs, and operational behavior; experimentation attaches quality signals and uses them to decide what to change. Together, they make it possible to compare quality with cost. section 1.4 section 7.1.3
Figure 8.1 The improvement loop. Production traffic generates traces. Scores (automated, LLM-based, and human) attach quality signals. Low-scoring traces flow into evaluation datasets. Offline evals compare prompt versions. Winning versions enter A/B tests. Results feed back into production. This continuous cycle turns static AI deployments into systems that improve over time.

The improvement loop connects production traces and scores to datasets, offline evaluation, and A/B tests. section 8.0
A service contract groups the improvement lifecycle into six concerns:
- Target lifecycle: register prompt versions, model configurations, or retrieval configurations; evaluate and promote them through tracked states. section 8.1 section 8.2
- Dataset management: combine curated test cases with production traces, including low-scoring or edge-case interactions. section 8.1 section 8.6
- Offline evaluation: run targets against datasets, score outputs, and compare candidates before deployment. section 8.1 section 8.3
- Online evaluation: apply automated and LLM-based scoring rules to a configurable sample of production traffic. section 8.1 section 8.3
- A/B testing: allocate live traffic to variants, record outcomes, and track sample sizes and statistical significance. section 8.1 section 8.4
- Human review: route selected traces to annotation queues to calibrate automated scorers and create ground-truth datasets. section 8.1 section 8.5
Targets reference artifacts stored by their home services rather than duplicating them. The experimentation service adds lifecycle status, change descriptions, and evaluation summaries. A target moves from draft to testing and, if successful, active; promotion deprecates the previous active target, while a failed evaluation returns the candidate to draft. section 8.2 section 8.2
Testing a patient-intake prompt
Register a new prompt version in the Model Service, then register it as an experiment target with a description of the change. Compare it with the existing version offline. For a live test, assign each patient consistently to a variant, use its prompt version, and record outcomes such as quality, turns to resolution, and cost. If no experiment is active, the workflow uses the default active prompt. section 8.2 section 8.4
Online scoring runs asynchronously after the response. Completed traces receive scores, and experiment outcomes are associated with assigned variants. Human annotations flow back into the score store, supporting scorer calibration and future offline datasets. section 8.5 section 8.5
