Chapter 8: The Experimentation Service: improving what your AI does
This chapter introduction positions the Experimentation Service as the infrastructure that turns observability signals into a continuous, controlled loop for improving AI behavior.
After this chapter you will be able to
- Continuous improvement loop. Explain the improvement loop that connects production traces, quality scores, low-scoring examples, evaluation datasets, offline evaluation, and production A/B testing. Describe that the loop repeats as validated production results create the next set of signals and problems to improve.
- Experimentation service purpose. Describe how the Experimentation Service differs from the Observability Service: observability answers what is happening, while experimentation answers what should change. State that it supplies systematic-improvement infrastructure including versioned prompts, evaluation datasets, scoring, A/B tests, and human review queues.
- Evidence-based experimentation. Justify controlled evaluation and statistical validation instead of changing prompts based on complaints or collecting telemetry without acting on it. Explain that datasets catch regressions, scoring measures changes, and statistical A/B tests establish whether a production change helped.
- Improvement lifecycle contract. Describe the six concerns covered by the Experimentation Service contract and how they form a progression from defining a change through evaluating it with automated, production, statistical, and human feedback.
- Targets and evaluation stages. Explain that targets are the changeable artifacts being tested—such as prompt versions, model configurations, or retrieval configurations—and distinguish dataset-based offline evaluation from sampled live-traffic online evaluation.
- Statistical and human feedback. Describe how A/B testing uses consistent user assignment plus sample-size and significance tracking to compare variants, and how annotation queues collect reviewer labels to create ground truth, calibrate automated scorers, and curate datasets.
- Experiment target scope. Explain which AI changes qualify as experiment targets and the common improvement lifecycle they follow: draft a change, evaluate it, promote an improvement, and deprecate the old version.
- Artifact ownership and metadata. Describe how an ExperimentTarget references an artifact by name and version in its home service rather than duplicating its content, while the Experimentation Service stores lifecycle, evaluation, and change metadata.
- Target state transitions. Trace the four-state target lifecycle: new targets begin in DRAFT, evaluation moves them to TESTING, a passing evaluation can be promoted to ACTIVE while the former active target becomes DEPRECATED, and a failed evaluation returns the target to DRAFT.
- Three-mode evaluation strategy. Describe how offline evaluation, online evaluation, and human evaluation work together to assess AI target changes before and after promotion.
- Promotion decision role. Explain that evaluation is the mechanism used to decide whether a change to a target deserves promotion from draft to production.
- Strengths of evaluation modes. Distinguish the specific role of each evaluation mode: offline evaluation catches regressions before deployment, online evaluation finds issues missed by curated datasets, and human evaluation supplies ground truth for the other modes.
- Evaluation dataset composition. Describe an evaluation dataset as a collection of test cases containing inputs, expected outputs, and metadata, and explain its role as an AI equivalent of a traditional software test suite.
- Curated and production cases. Explain why evaluation datasets combine hand-curated cases for known scenarios with production traces for unanticipated failures, edge cases, and changing user behavior.
- Production trace review pipeline. Trace how low-helpfulness traces and guardrail-triggered traces are selected from the Observability Service and added to a dataset only after required human review; explain why review prevents nonsensical user inputs or other garbage data from becoming test cases.
- Offline evaluation pipeline. Describe how offline evaluation runs every test case in a dataset through each target configuration before deployment, scores each generated response with configured metrics, and returns a side-by-side comparison.
- Repeated end-to-end evaluation. Explain why the pipeline runs each test case multiple times and averages its scores, and describe how retrieval can be included to evaluate a full RAG pipeline end to end.
- Relative judge-score interpretation. Explain that LLM-as-judge scores are comparative signals between variants rather than absolute measures of quality, and state why uncalibrated absolute scores should not be used as deployment gates.
- Purpose of online evaluation. Explain how online evaluation complements offline evaluation by evaluating sampled production traffic and revealing quality problems that curated test datasets can miss.
- Asynchronous trace scoring flow. Describe the online scoring flow: after a production trace completes, a scoring rule selects it, runs configured scorers asynchronously, attaches scores to the trace, and makes them available for dashboards, queries, and alerts.
- Annotation queue purposes. Explain the three purposes of annotation queues: obtaining ground-truth scores from production samples to calibrate automated scorers, surfacing edge cases those systems miss, and creating labeled data for evaluation datasets.
- Reinforcing evaluation cycle. Trace how offline evaluation, online evaluation, and human evaluation reinforce one another: offline evaluation catches regressions before deployment, online evaluation monitors production quality, and human evaluation calibrates scorers while building better datasets.
- Live traffic A/B experiments. Explain why A/B testing evaluates variants on real traffic rather than relying only on offline test datasets, and describe the parts of an experiment: variants, traffic allocation, and success metrics.
- Consistent variant assignment. Describe how consistent hashing combines an experiment name and patient ID to assign a deterministic variant, so the same patient receives the same variant throughout an experiment.
- Non-invasive workflow integration. Trace the three workflow additions for experimentation: assign_variant at the start, a conditional system-prompt-version override, and record_outcome at the end; explain what happens when no experiment is active.
- Experiment-aware workflow integration. Describe how a developer uses the Experimentation Service from workflow code: passes trace_context through normal service calls, optionally receives an experiment variant assignment, and records an outcome at the end.
- Asynchronous improvement loop. Trace the post-response improvement loop: completed production traces are asynchronously scored and written to the Score Store, experiment outcomes are correlated with assigned variants, and repeated production traffic, evaluation, and comparison improve alternatives over time.
What it builds on
- Quality scores and cost attribution (chapter 7) · Quality scores and cost attribution attached to AI activity.
- Putting it all together (chapter 7) · The integrated observability picture that supplies production traces and signals.
- Why AI systems need specialized observability (chapter 7) · Why AI systems need observability beyond infrastructure health.
- Scores: measuring response quality (chapter 7) · Quality scores as a way to measure response quality.
- Observability: Cost tracking and metrics (chapter 3) · Model-service metrics and cost tracking establish that AI behavior can be measured.
- How platform services report telemetry (chapter 7) · Platform services reporting telemetry provide the production traces used in evaluation.
- How the primitives connect: tracing a single request (chapter 7) · Request-level tracing provides the traces that can be sampled for review.
- Managing system prompts (chapter 3) · System prompts are managed as named, versioned artifacts.
Section by section
Introduction
This chapter introduction positions the Experimentation Service as the infrastructure that turns observability signals into a continuous, controlled loop for improving AI behavior.
- Continuous improvement loop: Explain the improvement loop that connects production traces, quality scores, low-scoring examples, evaluation datasets, offline evaluation, and production A/B testing. Describe that the loop repeats as validated production results create the next set of signals and problems to improve.
Why it matters: This loop replaces one-off AI changes with evidence-driven, repeatable improvement. - Experimentation service purpose: Describe how the Experimentation Service differs from the Observability Service: observability answers what is happening, while experimentation answers what should change. State that it supplies systematic-improvement infrastructure including versioned prompts, evaluation datasets, scoring, A/B tests, and human review queues.
Why it matters: Separating visibility from controlled improvement helps teams turn findings into reliable changes. - Evidence-based experimentation: Justify controlled evaluation and statistical validation instead of changing prompts based on complaints or collecting telemetry without acting on it. Explain that datasets catch regressions, scoring measures changes, and statistical A/B tests establish whether a production change helped.
Why it matters: Evidence-based experiments prevent teams from shipping unmeasured changes or leaving useful signals unused.
Key terms: improvement loop, production traffic, traces, quality signals, evaluation datasets, offline evaluation, Experimentation Service, Observability Service, systematic improvement, versioned prompts
8.1 The Experimentation Service contract
Defines the Experimentation Service’s gRPC contract as twenty operations organized around a lifecycle for safely measuring and improving AI behavior.
- Improvement lifecycle contract: Describe the six concerns covered by the Experimentation Service contract and how they form a progression from defining a change through evaluating it with automated, production, statistical, and human feedback.
Why it matters: The contract turns AI improvement from ad hoc prompt changes into a managed, repeatable lifecycle. - Targets and evaluation stages: Explain that targets are the changeable artifacts being tested—such as prompt versions, model configurations, or retrieval configurations—and distinguish dataset-based offline evaluation from sampled live-traffic online evaluation.
Why it matters: Separating a proposed target from the evidence used to assess it lets teams test changes before exposing users to them and continue monitoring afterward. - Statistical and human feedback: Describe how A/B testing uses consistent user assignment plus sample-size and significance tracking to compare variants, and how annotation queues collect reviewer labels to create ground truth, calibrate automated scorers, and curate datasets.
Why it matters: Reliable variant decisions need both statistical comparison and human judgment instead of relying on inconsistent user experiences or unverified automated scores.
Key terms: Experimentation Service, gRPC contract, improvement lifecycle, target lifecycle, dataset management, offline evaluation, target, prompt version, model configuration, retrieval configuration
8.2 Target lifecycle and evaluation
This section defines experiment targets as references to versioned AI artifacts and explains how the Experimentation Service evaluates and promotes them through a shared lifecycle.
- Experiment target scope: Explain which AI changes qualify as experiment targets and the common improvement lifecycle they follow: draft a change, evaluate it, promote an improvement, and deprecate the old version.
Why it matters: A common target model lets teams improve prompts, models, and retrieval settings using the same disciplined process. - Artifact ownership and metadata: Describe how an ExperimentTarget references an artifact by name and version in its home service rather than duplicating its content, while the Experimentation Service stores lifecycle, evaluation, and change metadata.
Why it matters: Separating artifact ownership from experimentation metadata prevents duplicated storage while preserving the evidence behind version decisions. - Target state transitions: Trace the four-state target lifecycle: new targets begin in DRAFT, evaluation moves them to TESTING, a passing evaluation can be promoted to ACTIVE while the former active target becomes DEPRECATED, and a failed evaluation returns the target to DRAFT.
Why it matters: The lifecycle makes promotion and rollback decisions explicit, repeatable, and auditable rather than informal deployment choices. - Side-by-side evaluation evidence: Describe how running an evaluation compares a candidate with the current active target across every dataset test case, streams progress, and produces per-metric results that teams use to judge quality tradeoffs.
Why it matters: A measured comparison exposes tradeoffs, so teams can make promotion decisions from evidence rather than guesses.
Key terms: experiment target, prompt version, base model, fine-tuned variant, LoRA adapter, retrieval configuration, ExperimentTarget, name, version, Model Service
8.3 Evaluation: datasets, offline scoring, online scoring, and human review
This section introduces three complementary evaluation modes that determine whether AI target changes should move from draft to production.
- Three-mode evaluation strategy: Describe how offline evaluation, online evaluation, and human evaluation work together to assess AI target changes before and after promotion.
Why it matters: Using all three modes gives teams evidence from controlled tests, real traffic, and human judgment rather than trusting a single signal. - Promotion decision role: Explain that evaluation is the mechanism used to decide whether a change to a target deserves promotion from draft to production.
Why it matters: A disciplined promotion decision prevents prompt, model, or retrieval changes from reaching production without evidence that they improve the system. - Strengths of evaluation modes: Distinguish the specific role of each evaluation mode: offline evaluation catches regressions before deployment, online evaluation finds issues missed by curated datasets, and human evaluation supplies ground truth for the other modes.
Why it matters: Knowing each mode's blind spots helps teams choose complementary evidence and avoid false confidence in automated scores.
Key terms: offline evaluation, online evaluation, human evaluation, annotation queues, curated datasets, live production traffic, evaluation, promotion, target lifecycle, draft
8.3.1 Datasets: the foundation of evaluation
This section explains how to build and maintain evaluation datasets by combining curated test cases with human-reviewed production failures and edge cases.
- Evaluation dataset composition: Describe an evaluation dataset as a collection of test cases containing inputs, expected outputs, and metadata, and explain its role as an AI equivalent of a traditional software test suite.
Why it matters: A useful evaluation dataset provides the cases needed to detect whether an AI system improves or regresses. - Curated and production cases: Explain why evaluation datasets combine hand-curated cases for known scenarios with production traces for unanticipated failures, edge cases, and changing user behavior.
Why it matters: Hand-written cases cover expected behavior, while production-derived cases reveal the real failures that matter to users. - Production trace review pipeline: Trace how low-helpfulness traces and guardrail-triggered traces are selected from the Observability Service and added to a dataset only after required human review; explain why review prevents nonsensical user inputs or other garbage data from becoming test cases.
Why it matters: Filtering and review turn production telemetry into trustworthy evaluation cases rather than polluting the dataset with misleading examples. - Evaluation dataset refresh cycle: State that evaluation datasets should be updated every few weeks with new production samples because user questions, seasonal topics, and unanticipated edge cases change over time.
Why it matters: Regular refreshes keep the dataset able to detect the regressions that users currently encounter.
Key terms: evaluation datasets, test cases, inputs, expected outputs, metadata, hand-curated test cases, production traces, edge cases, quality scores, helpfulness
8.3.2 Offline evaluation: comparing variants before deployment
Offline evaluation compares experiment targets on a dataset before deployment, using repeated scored runs and detailed results to guide safer iteration.
- Offline evaluation pipeline: Describe how offline evaluation runs every test case in a dataset through each target configuration before deployment, scores each generated response with configured metrics, and returns a side-by-side comparison.
Why it matters: This lets teams compare proposed AI changes before exposing production users to them. - Repeated end-to-end evaluation: Explain why the pipeline runs each test case multiple times and averages its scores, and describe how retrieval can be included to evaluate a full RAG pipeline end to end.
Why it matters: Repeated, end-to-end testing makes comparisons more reliable when model outputs vary and prompt changes interact with retrieved context. - Aggregate and case-level results: Distinguish aggregate scores from per-test-case breakdowns, and explain how a target can improve overall while regressing for a specific category of questions.
Why it matters: The breakdown identifies where a change succeeds or fails, giving the team a concrete direction for the next iteration. - Relative judge-score interpretation: Explain that LLM-as-judge scores are comparative signals between variants rather than absolute measures of quality, and state why uncalibrated absolute scores should not be used as deployment gates.
Why it matters: Treating a judge score as an objective percentage can lead to unsafe or unjustified deployment decisions.
Key terms: offline evaluation, experiment targets, dataset, test case, configured metrics, quality scores, non-deterministic, RAG pipeline, retrieval, end to end
8.3.3 Online evaluation: scoring production traffic in real-time
Online evaluation continuously scores sampled production traces asynchronously, exposing real-world quality gaps without delaying users.
- Purpose of online evaluation: Explain how online evaluation complements offline evaluation by evaluating sampled production traffic and revealing quality problems that curated test datasets can miss.
Why it matters: Production users bring unexpected inputs and language, so strong benchmark results alone do not prove that an AI application works well in reality. - Asynchronous trace scoring flow: Describe the online scoring flow: after a production trace completes, a scoring rule selects it, runs configured scorers asynchronously, attaches scores to the trace, and makes them available for dashboards, queries, and alerts.
Why it matters: Asynchronous scoring provides continuous quality visibility without adding latency to the user's response.
Key terms: online evaluation, online scoring, production traces, offline evaluation, test datasets, scoring rules, configured scorers, asynchronously, scores, dashboards
8.3.4 Annotation queues: calibrating with human judgment
This section explains how annotation queues bring human judgments into evaluation to calibrate automated scoring, catch missed edge cases, and improve datasets in a reinforcing evaluation cycle.
- Annotation queue purposes: Explain the three purposes of annotation queues: obtaining ground-truth scores from production samples to calibrate automated scorers, surfacing edge cases those systems miss, and creating labeled data for evaluation datasets.
Why it matters: Human review supplies the reference point that makes scalable automated evaluation trustworthy. - Human review workflow: Describe what a reviewer sees and produces when processing an annotation-queue item: a trace visualization with its input, output, spans, and existing automated scores, followed by rubric-based annotations that become scores with source=HUMAN.
Why it matters: Capturing human judgments as comparable scores lets the platform measure whether automated evaluation remains aligned with people. - Reinforcing evaluation cycle: Trace how offline evaluation, online evaluation, and human evaluation reinforce one another: offline evaluation catches regressions before deployment, online evaluation monitors production quality, and human evaluation calibrates scorers while building better datasets.
Why it matters: Using all three modes prevents quality evaluation from depending on only pre-release tests, production automation, or scarce human review.
Key terms: annotation queues, ground truth, production traces, automated scorers, LLM judges, evaluation datasets, trace visualization, input, output, spans
8.4 A/B testing infrastructure
This section explains how the Experimentation Service runs stable A/B tests on live traffic, integrates them into workflows with minimal changes, and reports statistically meaningful variant tradeoffs.
- Live traffic A/B experiments: Explain why A/B testing evaluates variants on real traffic rather than relying only on offline test datasets, and describe the parts of an experiment: variants, traffic allocation, and success metrics.
Why it matters: Real users behave in ways even well-curated test datasets cannot fully anticipate. - Consistent variant assignment: Describe how consistent hashing combines an experiment name and patient ID to assign a deterministic variant, so the same patient receives the same variant throughout an experiment.
Why it matters: Stable assignment prevents users from receiving jarringly different behavior across their interactions. - Non-invasive workflow integration: Trace the three workflow additions for experimentation: assign_variant at the start, a conditional system-prompt-version override, and record_outcome at the end; explain what happens when no experiment is active.
Why it matters: A workflow can support experiments without requiring a separate experiment-specific model path or changing its normal behavior. - Statistical experiment conclusions: Describe the statistical information used to compare variants—sample sizes, mean scores with confidence intervals, p-values, effect sizes, and required minimum sample size—and explain why the platform reports tradeoffs rather than choosing for product teams.
Why it matters: Statistical evidence helps teams avoid treating noise as improvement while making acceptable quality-versus-cost tradeoffs consciously.
Key terms: A/B testing, real traffic, experiment variants, traffic allocation, success metrics, offline evaluation, consistent hashing, patient ID, deterministic bucket assignment, variant assignment
8.5 Putting it all together
This section connects the Experimentation Service, observability telemetry, human review, and asynchronous scoring into a continuous production improvement loop.
- Experiment-aware workflow integration: Describe how a developer uses the Experimentation Service from workflow code: passes trace_context through normal service calls, optionally receives an experiment variant assignment, and records an outcome at the end.
Why it matters: This lets experimentation be added to an AI workflow without changing its basic service-call structure. - Human annotation feedback: Explain how annotation queues send selected traces to human reviewers and how their annotations return through the Observability Service to the Score Store for scorer calibration and ground-truth offline-evaluation datasets.
Why it matters: Human judgment makes automated quality scoring more trustworthy and supplies reliable examples for offline evaluation. - Asynchronous improvement loop: Trace the post-response improvement loop: completed production traces are asynchronously scored and written to the Score Store, experiment outcomes are correlated with assigned variants, and repeated production traffic, evaluation, and comparison improve alternatives over time.
Why it matters: Asynchronous evaluation lets the platform learn from real production interactions without delaying the response returned to the user.
Key terms: trace_context, experiment assignment, outcome recording, ExperimentClient, platform.experiments, lazy initialization, annotation queues, human reviewers, low-confidence LLM judge scores, guardrail-triggered requests
8.6 Summary
This summary ties together the Experimentation Service’s target lifecycle, evaluation loop, A/B testing safeguards, and low-intrusion developer workflow for continuously improving AI behavior.
Where this is used later
- Making Claw better: experimentation (chapter 10) · It supports improving the Claw assistant through experimentation.
- Streaming the response (chapter 10) · Evaluation of a target's behavior, including safety-relevant behavior.
- Seeing what Claw does: observability (chapter 10) · Online evaluation for scoring production traffic in real time.
- The blueprint (chapter 10) · How experimentation capabilities fit together.
- Summary (chapter 10) · how experimentation capabilities combine
Concepts explained in this chapter
- experimentation-as-a-service platformAn experimentation service is a platform service that provides infrastructure for running controlled experiments and measuring the effects of application changes.
- A/B testA controlled experiment that compares alternative variants by assigning them to different user or traffic groups and measuring outcomes.
- offline evaluationOffline evaluation assesses an AI system or change using previously collected datasets rather than live interactions.
- annotation queueAn annotation queue is an ordered collection of items awaiting human labeling or review.
- online evaluationOnline evaluation assesses an AI system or change using live production interactions and their outcomes.
- experiment targetAn experiment target is a record that references a versioned prompt, model configuration, or retrieval configuration and tracks its evaluation and lifecycle metadata.
- target lifecycleA target lifecycle is the progression of an experiment target through draft, testing, active, and deprecated states based on evaluation outcomes.
- continuous improvementAn asynchronous improvement loop evaluates completed interactions after serving them and feeds the resulting scores into systematic system improvement.
- evaluation datasetAn evaluation dataset is a collection of test cases containing inputs, expected outputs, and metadata for assessing an AI system or change.
- human-in-the-loopHuman review is the inspection and validation of an item by a person before it is accepted for an intended purpose.
- labelA ground truth label is a reference judgment assigned to an example output, typically by human annotation, for evaluating other scoring methods.
- experimentationAn experiment is a controlled procedure that assigns alternatives to traffic and measures their outcomes.
- gRPC contractA gRPC contract is a formal service interface that defines the operations and interaction structure exposed between gRPC services.
- qualityA score is a numerical or categorical value representing how well an observed operation or output performed against an evaluation criterion.
- evaluationEvaluation is the systematic assessment of an AI system or change against defined evidence and criteria.
- LLM-as-a-judgeAn LLM judge is a language model used to evaluate another model's output against specified criteria.
- automated scoringAn automated scorer is a non-generative evaluation procedure that computes response-quality scores from recorded metrics or rules.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
