Segmented evaluation compares model performance across defined user or data subsets instead of relying on one aggregate result. It reveals weaknesses hidden by averages and tests whether offline or online gains generalize across real-world contexts. section 2.3.4 section 6.6
Use segmented evaluation as a practical validation loop:
- Define meaningful slices before evaluation, including user groups, geographies, time periods, languages, devices, product surfaces, and edge cases. section 2.3.4 section 6.6
- Compare the primary metric and guardrails within each segment, then check whether gains are consistent or concentrated in a narrow cohort. section 6.6 section 8.3.4
- Treat post-hoc discoveries as hypotheses for follow-up testing rather than automatic launch decisions. section 6.6
- When offline and production results diverge, inspect sampling drift; PSI can flag distribution shifts without proving that performance degraded. section 2.3.4
Averages hide uneven results
An experiment may appear neutral overall while improving outcomes for new users and mobile users but producing little benefit for existing or desktop users. Those segment results support a more precise decision: launch broadly, retest a targeted change, or stop investing in the model direction. section 6.6
Segmented evaluation is not a license to search every subgroup for a positive result. Interpret slices alongside the original hypothesis, exposure validity, metric consistency, effect size, and product or engineering significance. section 6.6 section 8.3.4
