4 Understanding what your users really want

 

This chapter covers

  • Recognizing indicators of weak understanding
  • Measuring chatbot understanding
  • Assessing your chatbot’s current state
  • Collecting and preparing log data to measure chatbot understanding
  • Interpreting initial log data

A good chatbot experience is generally associated with the chatbot identifying (understanding) what the user wants. This is one of the key metrics you will use to measure performance. Sometimes a chatbot is deployed and has great initial understanding (or at least “good enough” for a pilot program). Over time, though, you may notice that it is returning wrong answers. Maybe your users are complaining more, either directly to the chatbot (“That doesn’t answer my question!”) or in the form of survey responses. Engagement could be trending downward while abandonment trends upward. You may start hearing from the call center about escalations that should have been handled in the virtual assistant. These are all indications that your conversational solution might be suffering from weak understanding.

4.1 Fundamentals of understanding

4.1.1 The impact of weak understanding

4.1.2 What causes weak understanding?

4.1.3 How do we achieve understanding with traditional conversational AI?

4.1.4 How do we achieve understanding with generative AI?

4.2 How is understanding measured?

4.2.1 Measuring understanding for traditional (classification-based) AI

4.2.2 Measuring understanding for generative AI

4.2.3 Measuring understanding with direct user feedback

4.3 Assessing where you are today

4.3.1 Assessing your traditional (classification-based) AI solution

4.3.2 Assessing your generative AI solution

4.4 Obtaining and preparing test data from logs

4.4.1 Obtaining production logs

4.4.2 Guidelines for identifying candidate test utterances

4.4.3 Preparing and scrubbing data for use in iterative improvements

4.4.4 The annotation process

Summary