Blind testing

As explained in Effective Conversational AI

Blind testing evaluates a classifier on utterances that were not included in the data used to train it.

Blind testing evaluates a classifier with test utterances it has not seen in training. This reveals whether it can recognize users’ intents beyond memorized examples. section 4.2.1

A blind test can use:

• AI-generated utterances manufactured for testing. section 4.2.1
• Representative utterances collected from actual user logs. section 4.2.1
• Accuracy, recall, and precision scores that show where the classifier succeeds or becomes confused. section 4.2.1

Testing beyond memorized utterances

If a classifier was trained on one phrasing of an intent, blind testing checks whether it recognizes a new phrasing that expresses the same intent. A correct prediction supports the chatbot’s ability to deliver the intended response or invoke the correct process flow. section 4.2.1

Figure 4.5 K-fold cross validation and generated test data are suitable for situations where representative data is not available. Once a solution is deployed to production, representative blind test data will produce the most reliable measurement of your classifier’s ability to understand.

Compare k-fold cross validation and generated test data with representative blind test data, which provides the most reliable measurement after deployment. section 4.2.1

How Effective Conversational AI covers it

How deep it goes

Defines unseen test utterances, identifies data sources and metrics, and briefly distinguishes blind testing from k-fold cross validation.

Examples

Uses AI-generated test utterances and representative utterances drawn from actual user logs.

What it assumes

Assumes a classifier and training data while defining how unseen test utterances measure understanding.

What you can do after

Evaluate classifier accuracy, recall, precision, and confusion using utterances absent from the training set.

How it connects

Connects classifier performance with accuracy, precision, recall, training sets, test sets, and k-fold cross validation.

liveBookmanning.com

Read Effective Conversational AI on liveBook Get the book at manning.com

This explainer was generated by Manning's knowledge graph from the text of Effective Conversational AI. The section links point to the parts of the book it draws on.