A high conversation count shows demand, not whether users got what they needed. Effectiveness asks whether a chatbot achieves its intended goal while providing a positive, efficient experience. section 3.3.2
Containment is one high-level measure: a conversation is contained when the chatbot handles the query without a human. Divide contained conversations by total conversations to calculate the containment rate. It indicates performance, but does not by itself show why users struggle or whether every contained conversation succeeded. section 3.3.2
PharmaBot’s dashboard
The team found that 45% of conversations transferred to the call center. That figure matters to business costs, but understanding performance more deeply requires looking beyond transfers to conversation outcomes and where users encounter difficulty. section 3.3.2
Figure 3.9 Breaking down why conversations are not contained gives more insight into bot performance and shows you where your bot needs improvement. One way to achieve this is by using detailed outcome classifications, which define the specific results of chatbot interactions. These classifications categorize conversations based on their resolution, user experience, and next steps.
Detailed outcome classifications distinguish conversation results and can reveal where improvement is needed. section 3.3.2
