An annotation queue is an ordered collection of traces awaiting human labeling or review. It brings human judgment into the improvement loop, where reviewers produce ground-truth scores, expose missed edge cases, and identify items for evaluation datasets. section 8.3.4
Use an annotation queue to:
- Calibrate automated scorers and LLM judges against human scores. section 8.3.4
- Route low-confidence results, guardrail-triggered requests, and random production samples to reviewers. section 8.5 section 8.3.4
- Build labeled evaluation datasets from reviewed production traces. section 8.3.4
Queue configuration
A queue can define a rubric with categorical correctness, numeric helpfulness, and a Boolean add-to-dataset decision. Routing rules can select traces when helpfulness falls below 0.5, a guardrail tag appears, or a two-percent random sample is chosen. Reviewers inspect the trace, existing scores, and rubric before submitting annotations. section 8.3.4
Figure 8.1 The improvement loop. Production traffic generates traces. Scores (automated, LLM-based, and human) attach quality signals. Low-scoring traces flow into evaluation datasets. Offline evals compare prompt versions. Winning versions enter A/B tests. Results feed back into production. This continuous cycle turns static AI deployments into systems that improve over time.

Use the improvement loop to trace how production requests become reviewed labels, evaluation data, scorer calibration, and later experiment feedback. section 8.0
Human review complements offline and online evaluation rather than replacing them. Completed traces can be scored asynchronously, compared with experiment variants, and fed back into the evaluation cycle. section 8.5 section 8.3
