A chatbot can keep a conversation to itself without helping the user. It can also transfer a user to a person for a good reason. A conversation outcome records what actually happened, rather than treating every contained conversation as a success and every transfer as a failure. section 3.3.2
Useful outcome classifications distinguish:
- Automated resolution: the bot answers the query and the user is satisfied. section 3.3.2
- Intentional transfer: a handoff required by business rules can still fulfill the bot’s job, such as collecting information before a specialist takes over. section 3.3.2
- Failure outcomes: abandonment or escalation can signal that the user struggled. Track when the bot was not wanted separately from these failures. section 3.3.2 section 3.4.1
A misleading transfer count
Suppose a bot must hand off a request because SMS confirmation is unavailable in the user’s state. Calling that transfer a failure hides a successful, intended path. By contrast, a user asking for an agent after the bot misunderstands them points to a problem worth investigating. section 3.3.2
Figure 3.11 The detailed outcome model depicts conversation outcomes and outcome details aggregated over a set time period. This provides much greater insight into bot performance than a binary “contained or not contained” model.
The detailed outcome model shows how outcome categories and their details reveal more than a contained-versus-uncontained count. section 3.3.2
To find a fix, plot outcomes over time alongside the last step users reached. Read transcripts where failures rise: when asked whether an appointment was for vaccines or testing, some users answered “yes,” then abandoned or escalated after the bot failed to handle it. Outcome counts locate the trouble; the conversation reveals why. Feedback can add context, but silence is not proof of satisfaction—users may simply disconnect. section 3.4.1 section 3.3.2
