Sampling selects a subset of observations or traffic for evaluation or processing. The right subset depends on the question: an evaluation may need broad user coverage, known edge cases, or enough observations to support a statistically significant result. Sampling is part of evaluation design, alongside metrics and confidence estimation. section 4.3.3 section 2.2
For LLM-as-a-judge evaluation, choose examples that reflect the diversity of users and use cases. Sampling only easy or narrowly distributed examples can leave scores unrepresentative. Early evaluations may use a fixed curated set; as a product matures, live or refreshed data can better reflect current usage. section 9.4.2
Choose and record a sampling strategy that fits the evaluation goal:
- Sample randomly from production logs, focus on a user segment, include known edge cases, balance languages or markets, or use a fixed golden dataset. section 9.10.2
- Record the exact examples, time window, filters, and model outputs in a dataset snapshot. Also log the judge configuration, rubric, and calibration results so a run can be revisited and justified. section 9.10.2
- When an aggregate score changes unexpectedly, check the sampling strategy and snapshot first. A different segment, time window, or filter can move the score without a change in model quality. section 10.6.2
Sampling shadow traffic
Instead of duplicating every production request, send a subset to the shadow system and throttle it to control cost. Set the goal first: the sample might need to cover key user segments or specific edge cases, or support statistical significance. Duplicate requests add load to shared infrastructure, so plan capacity and prevent shadow requests from causing unintended state changes. section 4.3.3
Figure 4.2 illustrates how shadow traffic adds additional load to the supporting infrastructure such as feature database layer, metadata services, caches and user context lookups. This is one of the primary risks of shadow traffic that's worth over communicating so that the machine learning or platform infrastructure teams aren’t surprised by an increase in requests as a result of the shadow traffic evaluation.
Trace where shadow requests add infrastructure load when deciding how much traffic to sample. section 4.3.3
Repeated sampling has a different meaning in judge evaluation: run the same prompt, model, and input multiple times to observe variation. A panel aggregates repeated judgments into a verdict; a stability test asks whether those judgments are consistent enough to aggregate. For a stability test, start with five to ten runs per validation example and inspect score spread or how often the majority verdict holds. Consistency alone does not establish correctness. section 10.1 section 10.4.2






























