7 How accurate is your assistant?
This chapter covers:
- Collecting test data for your assistant
- Assessing the accuracy of your assistant
- Selecting the best accuracy metric(s) to use for your assistant
Virtual assistants make predictions based on the way they are trained. How can you tell if this training is working well? You shouldn’t release a virtual assistant into the wild if you don’t know how well it works! You need to be able to tell if you are making the assistant smarter or dumber when you change the way you train it.
FICTITIOUS INC wants to assess their virtual assistant’s accuracy will be when they put it into production. The best way to test a virtual assistant’s accuracy is to see how well it predicts intents in production. This poses an interesting conundrum. They don't want to go to production without reasonable accuracy, but they won’t know the assistant’s true accuracy until it is in production.
The best way to handle this conundrum is to train and test your assistant iteratively, as shown in Figure 1. The virtuous cycle of Gather Data, Train, Test, and Improve, provides a repeatable methodology to build and improve your virtual assistant. It’s ok for FICTITIOUS INC to start with imperfect data because they can continually improve it. You have to start somewhere!
Figure 1 The virtuous cycle of testing how accurate your assistant is.
