4 Understanding what your users really want
This chapter covers
- Recognizing indicators of weak understanding
- Measuring chatbot understanding
- Assessing your chatbot’s current state
- Collecting and preparing log data to measure chatbot understanding
- Interpreting initial log data
A good chatbot experience is generally associated with the chatbot identifying (understanding) what the user wants. This is one of the key metrics you will use to measure performance. Sometimes a chatbot is deployed and has great initial understanding (or at least “good enough” for a pilot program). Over time, though, you may notice that it is returning wrong answers. Maybe your users are complaining more, either directly to the chatbot (“That doesn’t answer my question!”) or in the form of survey responses. Engagement could be trending downward while abandonment trends upward. You may start hearing from the call center about escalations that should have been handled in the virtual assistant. These are all indications that your conversational solution might be suffering from weak understanding.