A threshold is a predefined cutoff for deciding whether a metric meets a requirement or needs attention. In evaluation, thresholds turn measurements into decisions: whether to accept a trade-off, investigate a live regression, or block a launch. section 2.3.2 section 7.6.2 section 10.4.4
Agree on thresholds with product managers, data scientists, and engineers before interpreting results. Some metrics can be traded against one another; others act as guardrails. A small relevance gain, for example, may not justify a meaningful regression in safety, privacy, fairness, latency, or policy compliance. section 2.3.2
Balance engagement and diversity
Start by measuring baseline recommendation diversity, using measures such as intra-list diversity or catalog coverage. Then set a minimum that prevents overly narrow results—for example, require at least three of the top ten recommendations to come from different genres. Test the cutoff offline and refine it with user feedback and A/B testing while pursuing higher engagement. section 3.3.2
Offline findings can also determine live watchpoints. Record each offline metric, its online counterpart, production computation, and alert threshold in a metric mirror table. Base alert thresholds on offline benchmarks and historical production variability to reduce false positives. An online click-rate proxy is not identical to offline precision: clicks also reflect position, presentation, intent, and exposure. section 7.6 section 7.6.2
Table 7.2 Example: Metric Mirror Table for Recommendation Model A/B Test
|
Offline Metric
|
Online Watch point
|
Thresholds
|
Target Value
|
|
Diversity index
|
1−∑pi2 computed daily from top-N categories in live logs
|
Drop > 0.05 vs. control for ≥ 3 days
|
≥ 0.70
|
|
Latency p95
|
p95(model_response_time_ms) aggregated per day
|
> 400ms
|
≤ 350ms
|
|
Bias score by segment
|
Mean(predicted_score[group]) difference between any two groups (absolute value)
|
Gap > 0.10 for any demographic segment
|
≤ 0.05
|
|
Precision@10
|
Top-10 click-rate proxy: clicks on items shown in the top 10 divided by top-10 impressions, aggregated daily
|
Drop > 0.02 vs. control
|
Match or exceed control
|
|
Fairness gap
|
min(segment_score)/max(segment_score)
|
Ratio < 0.85
|
≥ 0.90
|
Compare target values with alert thresholds for diversity, latency, bias, precision, and fairness in a recommendation experiment. section 7.6.2
Gate on judge stability
For an LLM judge, set an acceptable variance before using its scores to decide a launch. For example, a standard deviation above 0.5 on a five-point scale could block the decision, regardless of the average score. If stability fails the threshold, revisit the prompt, add reasoning-first structure, or use panel-based aggregation. section 10.4.4



















