liveBook · Manning

Sampling

As explained in AI Model Evaluation

The process of selecting a subset of observations or traffic from a larger population for analysis or processing.

Sampling selects a subset of observations or traffic for evaluation or processing. The right subset depends on the question: an evaluation may need broad user coverage, known edge cases, or enough observations to support a statistically significant result. Sampling is part of evaluation design, alongside metrics and confidence estimation. section 4.3.3 section 2.2

For LLM-as-a-judge evaluation, choose examples that reflect the diversity of users and use cases. Sampling only easy or narrowly distributed examples can leave scores unrepresentative. Early evaluations may use a fixed curated set; as a product matures, live or refreshed data can better reflect current usage. section 9.4.2

Choose and record a sampling strategy that fits the evaluation goal:

  • Sample randomly from production logs, focus on a user segment, include known edge cases, balance languages or markets, or use a fixed golden dataset. section 9.10.2
  • Record the exact examples, time window, filters, and model outputs in a dataset snapshot. Also log the judge configuration, rubric, and calibration results so a run can be revisited and justified. section 9.10.2
  • When an aggregate score changes unexpectedly, check the sampling strategy and snapshot first. A different segment, time window, or filter can move the score without a change in model quality. section 10.6.2

Sampling shadow traffic

Instead of duplicating every production request, send a subset to the shadow system and throttle it to control cost. Set the goal first: the sample might need to cover key user segments or specific edge cases, or support statistical significance. Duplicate requests add load to shared infrastructure, so plan capacity and prevent shadow requests from causing unintended state changes. section 4.3.3

Figure 4.2 illustrates how shadow traffic adds additional load to the supporting infrastructure such as feature database layer, metadata services, caches and user context lookups. This is one of the primary risks of shadow traffic that's worth over communicating so that the machine learning or platform infrastructure teams aren’t surprised by an increase in requests as a result of the shadow traffic evaluation.

Trace where shadow requests add infrastructure load when deciding how much traffic to sample. section 4.3.3

Repeated sampling has a different meaning in judge evaluation: run the same prompt, model, and input multiple times to observe variation. A panel aggregates repeated judgments into a verdict; a stability test asks whether those judgments are consistent enough to aggregate. For a stability test, start with five to ten runs per validation example and inspect score spread or how often the majority verdict holds. Consistency alone does not establish correctness. section 10.1 section 10.4.2

How AI Model Evaluation covers it

How deep it goes
Explains selection options, documentation, operational trade-offs, and repeated-run stability checks without prescribing a universal sample size.
Examples
Uses production-log samples, segment and edge-case selection, shadow requests, and repeated five-point judge scores.
What you can do after
Practice selecting and documenting evaluation samples, controlling shadow traffic, and checking whether judge outputs are stable.
How it connects
Connects sampling to offline evaluation, confidence estimation, shadow testing, LLM-as-a-judge, and stability testing.

Also explained in

Compare how 31 books explain sampling →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.