liveBook · Manning

Experiment target

As explained in Designing AI Systems

An experiment target is a record that references a versioned prompt, model configuration, or retrieval configuration and tracks its evaluation and lifecycle metadata.

An experiment target records a versioned prompt, model configuration, or retrieval configuration without duplicating the artifact stored by its home service. It adds authorship, change rationale, lifecycle state, evaluation results, timestamps, and extensible metadata for comparison and promotion. section 8.2

Use a target to manage the improvement loop:

  • Create a DRAFT target that references an existing artifact version. section 8.2
  • Move it to TESTING and evaluate it against an evaluation dataset. section 8.2
  • Return failed evaluations to DRAFT for iteration; promote successful targets to ACTIVE. section 8.2
  • Automatically deprecate the previous active target when a replacement becomes ACTIVE. section 8.2
  • Use evaluation summaries, metric scores, test counts, and change descriptions to compare versions and preserve an audit trail. section 8.2

Registering a prompt experiment

Register a new prompt version with the Model Service, then register an experiment target using its name and auto-incremented version. Record why the prompt changed—for example, adding instructions to confirm understanding and offer related topics—and attach metadata such as a tracking identifier. Evaluate the target alongside other versions before promoting it. section 8.2

Listing 8.2 Experiment target lifecycle tracking
message ExperimentTarget {
  string name = 1;                              #A
  int32 version = 2;
  TargetType type = 3;                          #B
  string author = 4;
  string change_description = 5;                #C
  google.protobuf.Timestamp created_at = 6;
  TargetStatus status = 7;
  EvaluationSummary evaluation_summary = 8;     #D
  map<string, string> metadata = 9;
}
 
enum TargetType {
  TARGET_TYPE_UNSPECIFIED = 0;                  #F
  PROMPT = 1;
  MODEL_CONFIG = 2;
  RETRIEVAL_CONFIG = 3;
}
 
enum TargetStatus {
  TARGET_STATUS_UNSPECIFIED = 0;                #F
  DRAFT = 1;
  TESTING = 2;
  ACTIVE = 3;
  DEPRECATED = 4;
}
 
message EvaluationSummary {
  double overall_score = 1;
  map<string, double> metric_scores = 2;        #E
  string evaluation_id = 3;
  int32 test_cases_evaluated = 4;
}

Inspect the protocol-style data model for artifact references, target types, lifecycle states, evaluation summaries, metric scores, and extensible metadata. section 8.2

How Designing AI Systems covers it

How deep it goes
Defines fields, enums, evaluation summaries, and lifecycle transitions, then connects targets to offline, online, and human evaluation.
Examples
Uses prompt revisions, model swaps, LoRA variants, provider changes, retrieval parameters, and patient-intake prompt registration.
What it assumes
Assumes artifact registries and evaluation datasets, while teaching how to layer lifecycle and evaluation metadata across services.
What you can do after
Operate/configure an improvement loop that evaluates variants, promotes successful changes, and preserves experiment history.
How it connects
Connects prompt versions, model configurations, retrieval configurations, evaluation datasets, online evaluation, A/B testing, and human annotation.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Designing AI Systems. The section links point to the parts of the book it draws on.