8 The Experimentation Service: improving what your AI does
This chapter covers
- Designing the Experimentation Service contract for the improvement lifecycle
- Managing experiment targets from draft through evaluation to production
- Building evaluation datasets from curated test cases and production traces
- Scoring production traffic online and calibrating with human annotation queues
- A/B testing with consistent assignment and statistical significance
The Observability Service we built in the previous chapter answers, "what is happening?" The Experimentation Service answers, "what should we change?" It provides infrastructure for systematic improvement: versioned prompts, managed evaluation datasets, offline evaluation pipelines, online scoring of production traffic, A/B testing with statistical significance, and annotation queues for human review. This is where the platform closes the improvement loop.
The key insight behind the Experimentation Service is that observability and improvement are not separate concerns, rather they’re two halves of a continuous cycle. Production traffic generates traces. Scores attach quality signals. Low-scoring traces reveal problems. Those problems feed into evaluation datasets. Offline evaluation compares potential fixes. A/B testing validates the best fix in production. And the cycle repeats. Figure 8.1 illustrates this loop.