An eigenvalue is the scalar scaling factor λ for an eigenvector v that keeps its direction under a square matrix A: Av = λv. The matrix changes the vector’s length, but not its direction. Eigenvalues therefore describe characteristic behaviors of linear transformations and help identify meaningful axes in high-dimensional data. section 2.13
Several structural facts make eigenvalues useful in machine learning:
- Nonzero eigenvectors satisfy (A − λI)v = 0, so eigenvalues are found from the characteristic equation det(A − λI) = 0. section 2.13
- Eigenvectors associated with unequal eigenvalues are linearly independent. section 2.13.1
- For symmetric matrices, eigenvectors associated with different eigenvalues are orthogonal. section 2.13.2
- A matrix with enough linearly independent eigenvectors can be diagonalized as A = SΛS⁻¹; for symmetric A, this becomes A = SΛSᵀ. section 2.15
Rotation and the unchanged axis
A rotation about the Z-axis leaves every point on that axis in place. The Z-axis is therefore an eigenvector direction with eigenvalue 1: the transformation preserves both direction and length there. More generally, every rotation matrix has eigenvalue 1, whose corresponding eigenvector identifies its axis of rotation. section 2.13 section 2.14.4
Eigenvalues also quantify directional variation. For a covariance matrix, the largest eigenvalue gives the maximum data spread, and its eigenvector gives the corresponding principal direction. Ordering eigenvectors by decreasing eigenvalue produces orthogonal principal axes; retaining axes with larger eigenvalues supports PCA-based dimensionality reduction and can suppress lower-variance noise. section 4.7 section 5.10
Listing 2.13 Eigenvalues and vectors
from torch import linalg as LA
A = torch.tensor([[0.707, 0.707, 0],
[-0.707, 0.707, 0], [0, 0, 1]]) #1
l, e = LA.eig(A) #2
print("Eigen values are {}".format(l))
print("Eigen vectors are {}".format(e.T)) #3
A PyTorch listing computes eigenvalues and eigenvectors with torch.linalg.eig for a concrete rotation matrix. section 2.13.3
