A neural network layer transforms inputs and serves as a building block for defining a model. PyTorch Module classes implement layers with different purposes; even a linear regression model can be described with an input layer and an nn.Linear prediction layer. section 2.2
Common layer-based design choices include:
- Combine linear layers with nonlinear activation layers to model nonlinear parts of data. section 2.2 section 6.0
- Arrange layers into fully connected, convolutional, or recurrent architectures. section 6.0
- Add normalization layers or skip connections when changing the architecture to improve training. section 6.0
Composing layers in PyTorch
A fully connected model can be assembled with nn.Sequential: flatten the input, apply nn.Linear and nn.Tanh layers, repeat hidden blocks, and finish with a linear classification layer. section 6.0
fc_model = nn.Sequential(
nn.Flatten(),
nn.Linear(D, n), nn.Tanh(), #1
*[nn.Sequential(nn.Linear(n, n),nn.Tanh()) for _ in range(5)], #2
nn.Linear(n, classes),
)
Inspect the complete PyTorch layer composition, including repeated hidden linear-and-activation blocks. section 6.0
