liveBook · Manning

User experience

As explained in AI Model Evaluation

User experience is the overall experience a user has when interacting with a product, including its usefulness, quality, and responsiveness.

User experience is the overall experience of interacting with a product. For an AI feature, it depends not only on model output but also on whether that output is useful, fits the interface and workflow, and arrives when users expect it. section 1.8.3 section 4.1.1

Offline metrics are necessary before launch, but they cannot establish the full user impact of a model. Diagnostic evaluations can expose gaps hidden by averages and suggest improvements to investigate. User studies and online experiments then help test whether those changes improve the product experience. section 1.8.3 section 3.1.3 section 6.0

Recommendations beyond average recall

A movie recommender may have acceptable overall recall while favoring popular blockbusters over niche genres. A diagnostic evaluation can reveal the skew and suggest a test: does increasing exposure to underrepresented content improve the experience? Validate that hypothesis with user feedback, an online experiment, or downstream metrics such as watch time and retention. section 3.1.3

Evaluate the experience at several levels:

  • Check fit with the interface. A voice assistant can answer accurately yet frustrate users if its response is too long or poorly timed. Offline length filters or interaction-log simulations can approximate usability, but they do not replace user studies. section 1.8.3
  • Measure real and perceived latency. A delay hidden by caching may not affect behavior, while a delay blocking a response can be immediately visible. Streaming the first token can make a chatbot feel faster even when generation continues. section 4.1.1
  • Check downstream effects. A higher click-through rate can coincide with shorter sessions, lower satisfaction, or higher latency. Online data shows how the model interacts with real users and product constraints. section 6.0

Treat an offline improvement as a hypothesis, not proof that the product is better. Online experiments can also help determine whether observed benefits justify the cost and complexity of launching a change. section 6.1.1

How AI Model Evaluation covers it

How deep it goes
Explains evaluation approaches and latency trade-offs while stopping short of building scalable engineering systems.
Examples
Uses a voice assistant, a movie recommender favoring blockbusters, and a streaming chatbot to illustrate experience gaps.
What you can do after
Evaluate user experience by connecting offline diagnostics to user studies, online experiments, and downstream metrics.
How it connects
Links user experience to offline evaluation, diagnostic evaluation, A/B testing, and latency.

Also explained in

Compare how 13 books explain user experience →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.