concept Dunn index in category R

This is an excerpt from Manning's book Machine Learning with R, the tidyverse, and mlr. Login to get full access to this book.

Using the Dunn index to evaluate clustering performance

The Dunn index is another internal cluster metric that quantifies the ratio between the smallest distance between points in different clusters, and the largest distance within any of the clusters, referred to as the cluster’s diameter (see figure 16.5). These can be any distance metric but are commonly the Euclidean distance.

Figure 16.5. The Dunn index quantifies the ratio between the smallest distance between cases in different clusters (left-side plot) and the largest distance within a cluster (right-side plot).

The intuition here is that if we maintain the same diameter of our clusters but move the closest pair apart, the Dunn index will get larger. Conversely, if we maintain the same distance between cluster centroids but shrink the diameter of the clusters (by making the clusters denser), the Dunn index will also increase. As such, the number of clusters resulting in the largest Dunn index is the one that results in the largest minimum distance between clusters and the smallest maximum distance between cases within a cluster.

Calculating the Dunn index

It’s not necessary for you to memorize the formula for the Dunn index. If you are interested, we can define the Dunn index as