5 Improving weak understanding for traditional AI

 

This chapter covers

  • Identifying the types of errors a classifier can make
  • Establishing a baseline of current classifier performance
  • Using data science methodologies to identify and prioritize improvements
  • Infusing your traditional AI with generated content to enhance understanding

In this chapter, we will demonstrate a methodical, iterative approach to improving the understanding of a classification-based conversational solution. This chapter builds on the concepts introduced in the previous chapter and uses the output produced by the final exercise in section 4.4 (where you created a test set with the golden intent assigned to each utterance in a format that can be used by your testing tool). Later in this chapter, we’ll explore how large language models can supplement intent-driven output responses to deliver a more robust experience. (If you’re looking for generative AI improvement techniques, feel free to skip ahead to the next chapter.)

We will start by building an improvement plan and identifying the types of errors your classifier may be committing. Next, we’ll iterate through seven improvement cycles to solve the various problems you might see in your own text classifier. Although data science techniques are used, you do not need to be a data scientist to extract meaningful insights about your data using the methodologies presented in this chapter.

5.1 Building your improvement plan

5.1.1 Identify problematic patterns in misunderstood utterances

5.1.2 Incremental improvements

5.1.3 Where to start: Identifying the biggest problems

5.2 Solving “wrong intent matched”

5.2.1 Improve recall for one intent

5.2.2 Improve precision for one intent

5.2.3 Improve the F1 score for one intent

5.2.4 Improve precision and recall for multiple intents

5.3 Solving “no intent matched”

5.3.1 Clustering utterances for new intents

5.3.2 When to stop adding intents

5.4 Supplementing traditional AI with generative content

5.4.1 Combining traditional and generative AI for an intent

5.4.2 Prompting to convey understanding

Summary