Principal component analysis (PCA) reduces a data vector x in D dimensions to z in L dimensions, where L < D. It chooses a projection that maximizes the variance retained in the lower-dimensional representation. This makes high-dimensional relationships easier to inspect and can reduce the space needed to store features. section 8.3.1 section 8.3
Let W be a D × L matrix with orthonormal columns, so WᵀW = I. For centered data, the projection is z = Wᵀx, and its reconstruction is x̂ = Wz. The empirical covariance matrix Σ describes variation in the original space. For one unit-length projection direction w, maximizing projected variance means maximizing wᵀΣw subject to wᵀw = 1. A Lagrange multiplier gives Σw = λw: the direction is a covariance eigenvector, and its projected variance is the corresponding eigenvalue. Choosing the L eigenvectors with the largest eigenvalues gives W. section 8.3.1
A from-scratch implementation follows the derivation:
- Compute the empirical covariance matrix from centered feature values. section 8.3.1
- Find its eigenvalues and eigenvectors, then sort them by decreasing eigenvalue. section 8.3.1
- Keep the first L eigenvectors and project the data onto them. The supplied NumPy implementation computes a centered covariance matrix but calls X.dot(eig_vecs) without explicitly centering X for that final projection. section 8.3.1
Listing 8.3 Principal component analysis
import numpy as np
import matplotlib.pyplot as plt
np.random.seed(42)
class PCA():
def __init__(self, n_components = 2):
self.n_components = n_components
def covariance_matrix(self, X, Y=None):
if Y is None:
Y = X
n_samples = np.shape(X)[0]
covariance_matrix = (1 / (n_samples-1)) * (X - X.mean(axis=0))
.T.dot(Y - Y.mean(axis=0))
return covariance_matrix
def transform(self, X):
Sigma = self.covariance_matrix(X)
eig_vals, eig_vecs = np.linalg.eig(Sigma)
idx = eig_vals.argsort()[::-1] #1
eig_vals = eig_vals[idx][:self.n_components]
eig_vecs = np.atleast_1d(eig_vecs[:,idx])[:, :self.n_components]
X_transformed = X.dot(eig_vecs) #2
return X_transformed
if __name__ == "__main__":
n = 20
d = 5
X = np.random.rand(n,d)
pca = PCA(n_components=2)
X_pca = pca.transform(X)
print(X_pca)
plt.figure()
plt.scatter(X_pca[:,0], X_pca[:,1], color='b', alpha=0.5)
plt.title("Principal Component Analysis"); plt.xlabel("X1");
plt.ylabel("X2")
plt.show()
The NumPy listing makes the covariance, eigenvector ordering, and projection steps inspectable. section 8.3.1
Compressing image-search features
A ResNet50 image-search pipeline encodes Caltech 101 images as 2048-dimensional vectors, then uses PCA to reduce them to 300 dimensions before retrieving neighbors by cosine similarity. The reduction serves storage efficiency within the search pipeline. section 10.2.2
