liveBook · Manning

Chapter 8: The Experimentation Service: improving what your AI does

What you will learn in chapter 8 of Designing AI Systems

This chapter introduction positions the Experimentation Service as the infrastructure that turns observability signals into a continuous, controlled loop for improving AI behavior.

After this chapter you will be able to

What it builds on

Section by section

Introduction

This chapter introduction positions the Experimentation Service as the infrastructure that turns observability signals into a continuous, controlled loop for improving AI behavior.

Key terms: improvement loop, production traffic, traces, quality signals, evaluation datasets, offline evaluation, Experimentation Service, Observability Service, systematic improvement, versioned prompts

8.1 The Experimentation Service contract

Defines the Experimentation Service’s gRPC contract as twenty operations organized around a lifecycle for safely measuring and improving AI behavior.

Key terms: Experimentation Service, gRPC contract, improvement lifecycle, target lifecycle, dataset management, offline evaluation, target, prompt version, model configuration, retrieval configuration

8.2 Target lifecycle and evaluation

This section defines experiment targets as references to versioned AI artifacts and explains how the Experimentation Service evaluates and promotes them through a shared lifecycle.

Key terms: experiment target, prompt version, base model, fine-tuned variant, LoRA adapter, retrieval configuration, ExperimentTarget, name, version, Model Service

8.3 Evaluation: datasets, offline scoring, online scoring, and human review

This section introduces three complementary evaluation modes that determine whether AI target changes should move from draft to production.

Key terms: offline evaluation, online evaluation, human evaluation, annotation queues, curated datasets, live production traffic, evaluation, promotion, target lifecycle, draft

8.3.1 Datasets: the foundation of evaluation

This section explains how to build and maintain evaluation datasets by combining curated test cases with human-reviewed production failures and edge cases.

Key terms: evaluation datasets, test cases, inputs, expected outputs, metadata, hand-curated test cases, production traces, edge cases, quality scores, helpfulness

8.3.2 Offline evaluation: comparing variants before deployment

Offline evaluation compares experiment targets on a dataset before deployment, using repeated scored runs and detailed results to guide safer iteration.

Key terms: offline evaluation, experiment targets, dataset, test case, configured metrics, quality scores, non-deterministic, RAG pipeline, retrieval, end to end

8.3.3 Online evaluation: scoring production traffic in real-time

Online evaluation continuously scores sampled production traces asynchronously, exposing real-world quality gaps without delaying users.

Key terms: online evaluation, online scoring, production traces, offline evaluation, test datasets, scoring rules, configured scorers, asynchronously, scores, dashboards

8.3.4 Annotation queues: calibrating with human judgment

This section explains how annotation queues bring human judgments into evaluation to calibrate automated scoring, catch missed edge cases, and improve datasets in a reinforcing evaluation cycle.

Key terms: annotation queues, ground truth, production traces, automated scorers, LLM judges, evaluation datasets, trace visualization, input, output, spans

8.4 A/B testing infrastructure

This section explains how the Experimentation Service runs stable A/B tests on live traffic, integrates them into workflows with minimal changes, and reports statistically meaningful variant tradeoffs.

Key terms: A/B testing, real traffic, experiment variants, traffic allocation, success metrics, offline evaluation, consistent hashing, patient ID, deterministic bucket assignment, variant assignment

8.5 Putting it all together

This section connects the Experimentation Service, observability telemetry, human review, and asynchronous scoring into a continuous production improvement loop.

Key terms: trace_context, experiment assignment, outcome recording, ExperimentClient, platform.experiments, lazy initialization, annotation queues, human reviewers, low-confidence LLM judge scores, guardrail-triggered requests

8.6 Summary

This summary ties together the Experimentation Service’s target lifecycle, evaluation loop, A/B testing safeguards, and low-intrusion developer workflow for continuously improving AI behavior.

Where this is used later

Concepts explained in this chapter

← Chapter 7: The Observability Service: seeing what your AI does Chapter 9: The Workflow Service: orchestrating and deploying AI applications →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.