10 Improving your assistant

Fictitious Inc. has deployed their conversational AI assistant to production, but they are not achieving the success metrics they outlined for the solution. The assistant was supposed to reduce the burden on other customer service channels, but these channels have not seen a significant reduction in user activity. Fictitious Inc. knows how to troubleshoot their traditional applications but does not know where to start troubleshooting their conversational AI assistant.

Fictitious Inc. needs to quickly drill down into why their assistant is not performing well. They need to find out if their conversational flow does not work for users, if the intent mapping they have done does not work, or if there is some other core problem with their assistant.

Fictitious Inc. is in good company. Deploying a conversational AI to production is not the end; it is only the beginning. Figure 10.1 demonstrates the continuous improvement in an assistant’s life cycle. Continuous improvement is broadly applicable in software projects, and it is especially applicable for AI assistants.

10.1 Using a success metric to determine where to start improvements

10.1.1 Improving the first flow to fix containment problems

10.1.2 Inspecting other process flows for containment problems

10.2 Analyzing the classifier to predict future containment problems

10.2.1 Representative baseline

10.2.2 Finding gaps in the training data

10.3 When and why to improve your assistant

10.3.1 You can’t fix everything at once

10.3.2 You can’t always predict how users will react

10.3.3 User needs will change

10.3.4 Not every problem is technical

Summary