User experience is the overall experience of interacting with a product. For an AI feature, it depends not only on model output but also on whether that output is useful, fits the interface and workflow, and arrives when users expect it. section 1.8.3 section 4.1.1
Offline metrics are necessary before launch, but they cannot establish the full user impact of a model. Diagnostic evaluations can expose gaps hidden by averages and suggest improvements to investigate. User studies and online experiments then help test whether those changes improve the product experience. section 1.8.3 section 3.1.3 section 6.0
Recommendations beyond average recall
A movie recommender may have acceptable overall recall while favoring popular blockbusters over niche genres. A diagnostic evaluation can reveal the skew and suggest a test: does increasing exposure to underrepresented content improve the experience? Validate that hypothesis with user feedback, an online experiment, or downstream metrics such as watch time and retention. section 3.1.3
Evaluate the experience at several levels:
- Check fit with the interface. A voice assistant can answer accurately yet frustrate users if its response is too long or poorly timed. Offline length filters or interaction-log simulations can approximate usability, but they do not replace user studies. section 1.8.3
- Measure real and perceived latency. A delay hidden by caching may not affect behavior, while a delay blocking a response can be immediately visible. Streaming the first token can make a chatbot feel faster even when generation continues. section 4.1.1
- Check downstream effects. A higher click-through rate can coincide with shorter sessions, lower satisfaction, or higher latency. Online data shows how the model interacts with real users and product constraints. section 6.0
Treat an offline improvement as a hypothesis, not proof that the product is better. Online experiments can also help determine whether observed benefits justify the cost and complexity of launching a change. section 6.1.1












