Dimensionality reduction transforms data into fewer features while retaining important information. It addresses strongly correlated features by replacing them with compact composite features, making machine-learning data easier to model and visualize. section 9.2.4 section 9.2.5
Apply principal component analysis (PCA) through a repeatable preparation workflow:
- Inspect feature relationships with a Pearson-correlation heat map and identify highly correlated columns. section 9.2.4
- Scale the training features with StandardScaler so each feature has mean zero and standard deviation one. section 9.2.5
- Plot cumulative explained variance and retain components near the curve’s leveling-off point. section 9.2.5
Reducing DNS-derived features
A DNS feature set contains strongly correlated measurements such as fqdn_length, first_fqdn_label_length, and max_fqdn_label_length. After scaling, PCA combines correlated features into independent components; selecting four components produces X_train_pca with PC1 through PC4, while plotting PC1 and PC2 supports visual inspection. section 9.2.4
Listing 9.17 Plotting the PCA cumulative sum of the explained variance
from sklearn.decomposition import PCA #1
import numpy as np #2
pca=PCA().fit(X_train_scaled) #3
plt.figure() #4
plt.plot(np.cumsum(pca.explained_variance_ratio_)) #5
plt.xlabel('Number of principal components') #6
plt.ylabel('Cumulative explained variance') #7
plt.show() #8
Use the cumulative explained-variance plot to compare component counts before choosing the reduced representation. section 9.2.5
