Confusion matrix

As explained in Effective Conversational AI

A confusion matrix is a tabular summary comparing a classifier's predicted categories with the authoritative categories for evaluation examples.

A confusion matrix compares predicted intents with the correct intents for evaluation utterances, exposing where a classifier selects the wrong category or misses the right one. section 5.1.1

Read each prediction through four possible outcomes:

• True positive: the prediction matches the correct intent. section 5.1.1
• False positive: the classifier selects an intent that is incorrect, creating over-selection. section 5.1.1
• False negative: the classifier fails to select the correct intent, creating under-selection. section 5.1.1
• True negative: the prediction does not match an incorrect intent. section 5.1.1

Request_Agent evaluation

Seven utterances include four correct Request_Agent predictions, two false positives, and one missed match. This produces 80% recall, 66% precision, and a 0.72 F1 score. section 5.1.1

Figure 5.1 In a 2 × 2 confusion matrix, the possible outcomes are derived by comparing the predicted intent to the actual intent.

Inspect the 2 × 2 layout that derives outcomes by comparing predicted and actual intents. section 5.1.1

How Effective Conversational AI covers it

How deep it goes

Defines four outcomes and works through recall, precision, and F1 calculations using an intent-level evaluation example.

Examples

Uses Request_Agent utterances such as “Speak with an agent,” “What can I ask you,” and “Somebody hit my car.”

What it assumes

Assumes labeled test utterances and classifier intents, while defining the prediction outcomes needed for evaluation.

What you can do after

Diagnose over-selection and under-selection by interpreting prediction outcomes and calculating intent-level evaluation metrics.

How it connects

Connects classifier evaluation with intents, recall, precision, F1 score, true positives, false positives, and false negatives.

liveBookmanning.com

Read Effective Conversational AI on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Starting Data Analytics with Generative AI and PythonA confusion matrix is a table comparing actual classes with predicted classes to summarize classification outcomes.
• 

  liveBookmanning.com
  Grokking Machine LearningA tabular representation of classification outcomes divided into true-positive, true-negative, false-positive, and false-negative counts.
• 

  liveBookmanning.com
  Statistics Slam DunkA confusion matrix is a table that cross-classifies predicted and observed classes to count correct and incorrect classifications.
• 

  liveBookmanning.com
  Statistics Every Programmer NeedsA confusion matrix is a table that summarizes classification results by cross-tabulating actual classes against predicted classes.
• 

  liveBookmanning.com
  Knowledge Graphs and LLMs in ActionA confusion matrix is a tabular summary of predicted and actual class assignments, including counts of correct and incorrect classifications.
• 

  liveBookmanning.com
  Natural Language Processing in Action, Second EditionA confusion matrix is a table counting predictions by their actual and predicted classes.
• 

  liveBookmanning.com
  Grokking StatisticsA confusion matrix is a table that cross-tabulates predicted classes against actual classes and counts correct and incorrect classifications.
• 

  liveBookmanning.com
  Machine Learning Algorithms in DepthA confusion matrix is a table counting predictions by combinations of predicted and actual classes.
• 

  liveBookmanning.com
  Graph Algorithms for Data ScienceA confusion matrix is a table counting classification outcomes by actual class and predicted class.
• 

  liveBookmanning.com
  Data-Driven CybersecurityA confusion matrix is a table that cross-classifies predicted classes against actual classes to show correct and incorrect classification outcomes.
• 

  liveBookmanning.com
  Deep Learning with PyTorch, Second EditionA 2D breakdown of classification outcomes into true/false positives/negatives used to analyze model predictions.
• 

  liveBookmanning.com
  First-Party Data ActivationA confusion matrix is a table that organizes the counts or shares of correct and incorrect outcomes for a classification system.
• 

  liveBookmanning.com
  Financial AI in PracticeA table that summarizes classification results by comparing predicted classes with actual classes.

Compare how 14 books explain confusion matrix →

Related concepts

• false positivehas part · A false positive is a result that incorrectly indicates a condition or class is present when it is absent.
• classifiermeasures · A classifier is a predictive computational model that assigns inputs to one or more discrete classes or categories.
• true positivehas part · A true positive is a positive identification or prediction that correctly indicates the presence of the condition, class, or item being sought.
• false negativehas part · A false negative is an outcome that incorrectly indicates a target condition is absent when it is actually present.
• true negativehas part · A true negative is a classification outcome in which an instance that actually belongs to the negative class is correctly predicted as negative.
• language modelmeasures · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.

This explainer was generated by Manning's knowledge graph from the text of Effective Conversational AI. The section links point to the parts of the book it draws on.