liveBook · Manning

Label

As explained in Designing AI Systems

A ground truth label is a reference judgment assigned to an example output, typically by human annotation, for evaluating other scoring methods.

A ground truth label is a human reviewer’s judgment assigned to an example output. It provides the reference against which automated scorers and LLM judges are calibrated. section 8.3.4 section 7.6.1

Labels usually come from annotation queues. Reviewers inspect the input, output, trace, existing scores, and rubric, then record judgments such as correctness, helpfulness, or dataset inclusion. section 8.3.4

Use labels to strengthen an evaluation system:

  • Sample production traces for high-accuracy human review. section 7.6.1
  • Compare human scores with automated and model-judge scores to detect calibration drift and systematic disagreement. section 8.3.4 section 7.6.1
  • Promote selected labeled traces into evaluation datasets for future offline testing. section 8.3.4 section 8.1

Annotation queue

A review queue can route outputs with helpfulness below 0.5, guardrail-triggered traces, and a random 2% sample to human reviewers. Its rubric can capture categorical correctness, numeric helpfulness, and whether the trace should enter the evaluation dataset. section 8.3.4

Listing 8.6 Annotation queue for human review
queue = platform.experiments.create_annotation_queue(
    name="intake-quality-review",
    workflow_id="patient-intake",
    rubric=[       #A
        {"name": "correctness", "type": "categorical",
         "options": ["correct", "partially_correct",
                    "incorrect"]},
        {"name": "helpfulness", "type": "numeric",
         "min": 0.0, "max": 1.0},
        {"name": "add_to_dataset", "type": "boolean",    #B
         "description": "Should this be added to the "
                       "eval dataset?"},
    ],
    routing_rules=[
        {"condition": "score.helpfulness < 0.5",
         "source": "model_judge"},     #C
        {"condition": "tag contains 'guardrail_triggered'"},
        {"condition": "random_sample", "rate": 0.02},   #D
    ],
    reviewers=["sarah@healthfirst.com",
               "dr.patel@healthfirst.com"],
)

Inspect the queue configuration to see how rubric fields, routing conditions, sampling, and reviewers turn human judgment into operational labels. section 8.3.4

Listing 7.11 Score data model and scorer implementations
message Score {
  string score_id = 1;
  string trace_id = 2;
  string span_id = 3;                           #A
  string generation_id = 4;
  string name = 5;                              #B
  oneof value {
    double numeric_value = 6;                   #C
    string categorical_value = 7;
    bool boolean_value = 8;
  }
  ScoreSource source = 9;                       #D
  string comment = 10;
  map<string, string> metadata = 11;
  google.protobuf.Timestamp timestamp = 12;
}
 
enum ScoreSource {
  SCORE_SOURCE_UNSPECIFIED = 0;                 #E
  AUTOMATED = 1;
  MODEL_JUDGE = 2;
  HUMAN = 3;
  USER_FEEDBACK = 4;
}

Inspect the score model to see how a human-produced value is attached to a trace and distinguished through its HUMAN source. section 7.6.1

How Designing AI Systems covers it

How deep it goes
Defines reviewer workflow, routing, rubric fields, score provenance, and calibration, but stops short of annotation-interface internals.
Examples
Uses correctness, helpfulness, dataset-inclusion decisions, production-trace samples, and comparisons against automated scores.
What it assumes
Assumes familiarity with production traces, scoring methods, evaluation datasets, and quality dimensions such as correctness and helpfulness.
What you can do after
Operate annotation queues to collect human scores, measure scorer agreement, identify missed edge cases, and curate evaluation data.
How it connects
Connects human evaluation with annotation queues, automated scoring, LLM-as-judge evaluation, calibration, and evaluation datasets.

Also explained in

Compare how 29 books explain label →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Designing AI Systems. The section links point to the parts of the book it draws on.