The silhouette coefficient measures how well each observation fits its assigned cluster by comparing cohesion within that cluster with separation from the nearest cluster. For a point, a is its mean intracluster distance and b is its mean distance from the nearest other cluster; the score is (b − a) / max(a, b). Dataset scores average the point scores and range from −1 to 1. section 8.4
Use silhouette analysis to assess candidate k values in a repeatable workflow:
- Evaluate each candidate k using average silhouette scores; higher values generally indicate denser, better-separated clusters. section 8.4
- Interpret scores near 1 as strong separation and scores near −1 as possible misassignment. section 8.4
- Compare silhouette results with the elbow method, especially when the elbow point is unclear, while accounting for substantially higher computation. section 8.4 section 8.7
Selecting clusters for network-connection analysis
A k-means investigation of network connections used the elbow method to select k while isolating small clusters associated with malicious beaconing. Silhouette analysis provides an alternative or confirmation: one run selected k = 5, although the highest plotted score occurred at k = 10, illustrating why candidate selection and plot interpretation should be compared. section 8.3.2 section 8.7
Listing 8.28 Applying Silhouette analysis to df_features_one_hot_enc
df_features_one_hot_enc = df_features_one_hot_enc.drop(['cluster'],\
axis='columns') #1
visualizer = KElbowVisualizer(model, k=(2,10), metric='Silhouette')
visualizer.fit(df_features_one_hot_enc)
visualizer.show()
plt.show()
Run silhouette analysis across k = 2 through 10 with Yellowbrick's KElbowVisualizer and inspect the resulting plot. section 8.4
