16 Testing and measuring performance in voice systems

 

This chapter covers

  • Measuring the metrics that matter for voice system performance
  • Testing and reporting system performance
  • Collecting valid user data
  • Discussion of additional system testing pre- and post-rollout

So you’ve built your voice system and applied everything you’ve learned. Now you’re ready for the rest of the world to flock to your implementation. How do you bring it to the world without it breaking in some horrid way? Will you know if people are successful interacting with your VUI or like talking to it? If you followed all the ‘rules,’ why do you even need to worry about any of this?

This chapter explores how to test voice systems and measure their performance in terms of both system accuracy and usability. The information and metrics you capture feed directly into your ability to improve system performance (chapter 17).

16.1  Testing voice system performance

All software needs testing; voice systems are no different. But the details of how you perform testing on a voice system, and across voice development platforms, differ from what you might be used to if you’ve tested non-voice interfaces and have their own special challenges.

16.1.1    Recognition testing

16.1.2    Dialog traversal: functional end-to-end testing

16.1.3    Wake-word and speech detection testing

16.1.4    Additional system integration testing

16.2  Testing usability and task completion

16.2.1    Voice usability testing concepts

16.2.2    Wizard of Oz studies

16.3  Tracking and measuring performance

16.3.1    Recognition performance metrics

16.3.2    Task completion metrics

16.3.3    User satisfaction metrics

16.4  What’s next?

16.5  Summary