A ground truth label is a human reviewer’s judgment assigned to an example output. It provides the reference against which automated scorers and LLM judges are calibrated. section 8.3.4 section 7.6.1
Labels usually come from annotation queues. Reviewers inspect the input, output, trace, existing scores, and rubric, then record judgments such as correctness, helpfulness, or dataset inclusion. section 8.3.4
Use labels to strengthen an evaluation system:
- Sample production traces for high-accuracy human review. section 7.6.1
- Compare human scores with automated and model-judge scores to detect calibration drift and systematic disagreement. section 8.3.4 section 7.6.1
- Promote selected labeled traces into evaluation datasets for future offline testing. section 8.3.4 section 8.1
Annotation queue
A review queue can route outputs with helpfulness below 0.5, guardrail-triggered traces, and a random 2% sample to human reviewers. Its rubric can capture categorical correctness, numeric helpfulness, and whether the trace should enter the evaluation dataset. section 8.3.4
Listing 8.6 Annotation queue for human review
queue = platform.experiments.create_annotation_queue(
name="intake-quality-review",
workflow_id="patient-intake",
rubric=[ #A
{"name": "correctness", "type": "categorical",
"options": ["correct", "partially_correct",
"incorrect"]},
{"name": "helpfulness", "type": "numeric",
"min": 0.0, "max": 1.0},
{"name": "add_to_dataset", "type": "boolean", #B
"description": "Should this be added to the "
"eval dataset?"},
],
routing_rules=[
{"condition": "score.helpfulness < 0.5",
"source": "model_judge"}, #C
{"condition": "tag contains 'guardrail_triggered'"},
{"condition": "random_sample", "rate": 0.02}, #D
],
reviewers=["sarah@healthfirst.com",
"dr.patel@healthfirst.com"],
)Inspect the queue configuration to see how rubric fields, routing conditions, sampling, and reviewers turn human judgment into operational labels. section 8.3.4
Listing 7.11 Score data model and scorer implementations
message Score {
string score_id = 1;
string trace_id = 2;
string span_id = 3; #A
string generation_id = 4;
string name = 5; #B
oneof value {
double numeric_value = 6; #C
string categorical_value = 7;
bool boolean_value = 8;
}
ScoreSource source = 9; #D
string comment = 10;
map<string, string> metadata = 11;
google.protobuf.Timestamp timestamp = 12;
}
enum ScoreSource {
SCORE_SOURCE_UNSPECIFIED = 0; #E
AUTOMATED = 1;
MODEL_JUDGE = 2;
HUMAN = 3;
USER_FEEDBACK = 4;
}Inspect the score model to see how a human-produced value is attached to a trace and distinguished through its HUMAN source. section 7.6.1




























