Principal component analysis (PCA) reduces a feature vector from D dimensions to D′ dimensions, then approximately reconstructs it. Its encoder is a weight matrix W; its decoder uses Wᵀ, making PCA a linear autoencoder with shared weights. The reconstruction objective uses squared error, while an orthogonality constraint keeps the projection well behaved. section 7.1.1
Implement PCA in PyTorch by mapping the mathematical pieces directly to modules:
- Create an encoder with nn.Flatten followed by nn.Linear(D, n, bias=False), where n is the bottleneck dimension. section 7.1.2
- Reuse the encoder's weights in a TransposeLinear module to implement Wᵀ; give this layer a separate bias only if one is needed. section 7.1.2
- Decode with the transposed layer and reshape the result back to the original input shape, such as (B, 1, 28, 28). section 7.1.2
- Train with mean squared reconstruction error plus a penalty comparing WWᵀ with the identity matrix; orthogonal initialization can support the intended constraint. section 7.1.2
From PCA to a nonlinear autoencoder
Add Tanh after a linear bottleneck, remove weight sharing, and use a separate linear decoder. Train with ordinary MSE, then inspect two-dimensional encodings and reconstructed images to compare artifacts and overlap. section 7.1.5
linearLayer = nn.Linear(D, n, bias=False) #1 pca_encoder = nn.Sequential( #2 nn.Flatten(), linearLayer, ) pca_decoder = nn.Sequential( #3 TransposeLinear(linearLayer, bias=False), View(-1, 1, 28, 28) #4 ) pca_model = nn.Sequential( #5 pca_encoder, pca_decoder )
Shows the compact encoder, shared-weight decoder, and reshaping pipeline used to express PCA as a PyTorch model. section 7.1.2
