Dimensionality reduction converts many numeric variables into a few summary dimensions, making complex tabular data easier to visualize and inspect for patterns or anomalies. section 2.4.3
Choose an approach according to the structure you need to preserve:
- PCA and SVD create dimensions through weighted linear combinations of features, making them useful for broadly linear structure. section 2.4.3
- t-SNE and UMAP can reveal nonlinear patterns, clusters, and isolated points, but their projections depend strongly on hyperparameters. section 2.4.3
- Treat a projection as an exploratory aid rather than a definitive representation: information loss can produce misleading or difficult-to-interpret plots. section 2.4.3
A practical t-SNE inspection
Select numeric and ordinal features, fill missing values with column means, fit a two-dimensional t-SNE projection, and plot the resulting points to inspect clusters and anomalies. section 2.4.3
Listing 2.9 Plotting a t-SNE low-dimensional projection of a dataset
from sklearn.manifold import TSNE #1
import matplotlib.pyplot as plt #2
tsne = TSNE(n_components=2,
perplexity=30.,
init="random",
learning_rate="auto",
random_state=42) #3
X = data[numeric_feats + ordinal_feats].fillna(
data[numeric_feats + ordinal_feats].mean()) #4
projection_2D = tsne.fit_transform(X) #5
plt.figure(figsize=(15, 15))
plt.scatter(projection_2D[:, 0], projection_2D[:, 1],
edgecolor='none',
alpha=0.80,
s=10)
plt.show() #6
Runnable scikit-learn code demonstrates two-dimensional t-SNE projection and visualization. section 2.4.3
