liveBook · Manning

Threshold

As explained in AI Model Evaluation

A threshold is a predefined cutoff value used to determine whether a metric or evaluation result satisfies a requirement.

A threshold is a predefined cutoff for deciding whether a metric meets a requirement or needs attention. In evaluation, thresholds turn measurements into decisions: whether to accept a trade-off, investigate a live regression, or block a launch. section 2.3.2 section 7.6.2 section 10.4.4

Agree on thresholds with product managers, data scientists, and engineers before interpreting results. Some metrics can be traded against one another; others act as guardrails. A small relevance gain, for example, may not justify a meaningful regression in safety, privacy, fairness, latency, or policy compliance. section 2.3.2

Balance engagement and diversity

Start by measuring baseline recommendation diversity, using measures such as intra-list diversity or catalog coverage. Then set a minimum that prevents overly narrow results—for example, require at least three of the top ten recommendations to come from different genres. Test the cutoff offline and refine it with user feedback and A/B testing while pursuing higher engagement. section 3.3.2

Offline findings can also determine live watchpoints. Record each offline metric, its online counterpart, production computation, and alert threshold in a metric mirror table. Base alert thresholds on offline benchmarks and historical production variability to reduce false positives. An online click-rate proxy is not identical to offline precision: clicks also reflect position, presentation, intent, and exposure. section 7.6 section 7.6.2

Table 7.2 Example: Metric Mirror Table for Recommendation Model A/B Test
Offline Metric
Online Watch point
Thresholds  
Target Value
Diversity index
1−∑pi2 computed daily from top-N categories in live logs
Drop > 0.05 vs. control for ≥ 3 days
≥ 0.70
Latency p95
p95(model_response_time_ms) aggregated per day
> 400ms
≤ 350ms
Bias score by segment
Mean(predicted_score[group]) difference between any two groups (absolute value)
Gap > 0.10 for any demographic segment
≤ 0.05
Precision@10
Top-10 click-rate proxy: clicks on items shown in the top 10 divided by top-10 impressions, aggregated daily
Drop > 0.02 vs. control
Match or exceed control
Fairness gap
min(segment_score)/max(segment_score)
Ratio < 0.85
≥ 0.90

Compare target values with alert thresholds for diversity, latency, bias, precision, and fairness in a recommendation experiment. section 7.6.2

Gate on judge stability

For an LLM judge, set an acceptable variance before using its scores to decide a launch. For example, a standard deviation above 0.5 on a five-point scale could block the decision, regardless of the average score. If stability fails the threshold, revisit the prompt, add reasoning-first structure, or use panel-based aggregation. section 10.4.4

How AI Model Evaluation covers it

How deep it goes
Explains how to establish baselines, agree on cutoffs, validate them, and account for production variability.
Examples
Uses recommendation diversity minimums, live latency alerts, and LLM-judge variance limits as concrete cutoffs.
What you can do after
Operate and configure evaluation thresholds for product trade-offs, live alerts, and judge-stability gates.
How it connects
Connects thresholds to offline evaluation, online monitoring, guardrails, A/B testing, and variance.

Also explained in

Compare how 20 books explain threshold →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.