liveBook · Manning

Experimentation-as-a-service platform

As explained in Designing AI Systems

An experimentation service is a platform service that provides infrastructure for running controlled experiments and measuring the effects of application changes.

An experimentation-as-a-service platform provides shared infrastructure for testing and improving AI applications. It replaces blind changes with versioned targets, quality measurement, controlled comparisons, and a path from evaluation to production. section 1.1.3 section 8.0

The service complements observability rather than replacing it. Observability records traces, costs, and operational behavior; experimentation attaches quality signals and uses them to decide what to change. Together, they make it possible to compare quality with cost. section 1.4 section 7.1.3

Figure 8.1 The improvement loop. Production traffic generates traces. Scores (automated, LLM-based, and human) attach quality signals. Low-scoring traces flow into evaluation datasets. Offline evals compare prompt versions. Winning versions enter A/B tests. Results feed back into production. This continuous cycle turns static AI deployments into systems that improve over time.

The improvement loop connects production traces and scores to datasets, offline evaluation, and A/B tests. section 8.0

A service contract groups the improvement lifecycle into six concerns:

  • Target lifecycle: register prompt versions, model configurations, or retrieval configurations; evaluate and promote them through tracked states. section 8.1 section 8.2
  • Dataset management: combine curated test cases with production traces, including low-scoring or edge-case interactions. section 8.1 section 8.6
  • Offline evaluation: run targets against datasets, score outputs, and compare candidates before deployment. section 8.1 section 8.3
  • Online evaluation: apply automated and LLM-based scoring rules to a configurable sample of production traffic. section 8.1 section 8.3
  • A/B testing: allocate live traffic to variants, record outcomes, and track sample sizes and statistical significance. section 8.1 section 8.4
  • Human review: route selected traces to annotation queues to calibrate automated scorers and create ground-truth datasets. section 8.1 section 8.5

Targets reference artifacts stored by their home services rather than duplicating them. The experimentation service adds lifecycle status, change descriptions, and evaluation summaries. A target moves from draft to testing and, if successful, active; promotion deprecates the previous active target, while a failed evaluation returns the candidate to draft. section 8.2 section 8.2

Testing a patient-intake prompt

Register a new prompt version in the Model Service, then register it as an experiment target with a description of the change. Compare it with the existing version offline. For a live test, assign each patient consistently to a variant, use its prompt version, and record outcomes such as quality, turns to resolution, and cost. If no experiment is active, the workflow uses the default active prompt. section 8.2 section 8.4

Online scoring runs asynchronously after the response. Completed traces receive scores, and experiment outcomes are associated with assigned variants. Human annotations flow back into the score store, supporting scorer calibration and future offline datasets. section 8.5 section 8.5

How Designing AI Systems covers it

How deep it goes
Defines a service contract and target lifecycle, then details evaluation modes, consistent variant assignment, and asynchronous feedback.
Examples
Uses a patient-intake prompt comparison to demonstrate version registration, evaluation, variant assignment, and outcome recording.
What you can do after
Build an experimentation service around target management, evaluation pipelines, production scoring, and controlled comparisons.
How it connects
Connects observability, distributed tracing, quality scoring, evaluation datasets, A/B testing, and human annotation.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Designing AI Systems. The section links point to the parts of the book it draws on.