A linear layer, also called a fully connected layer, connects every unit in one layer to every unit in the next. With m input neurons and n output neurons, it has mn weighted connections. It supplies the learned weighted transformation that moves information between stages of a neural network. section 8.3
Each output unit receives a weighted sum of outputs from the preceding layer. Weight notation places the destination unit before the source unit, which makes the collection of connections convenient to express with a weight matrix. A perceptron’s weighted sum and sigmoid can be treated as separate layers: the linear layer performs the weighted-sum stage, while the activation introduces nonlinearity. section 8.3 section 11.1
Linear layers are used at several points in a network:
- Between hidden representations, paired with sigmoid activations in a demand-prediction network. section 8.5
- As a classifier’s score-producing output, before softmax converts raw scores to class probabilities. section 9.2.3
- After convolutional feature extraction, producing class logits in LeNet. section 11.1
- As prediction heads that output class probabilities and bounding-box coordinates after RoI pooling in Faster R-CNN. section 11.4
A PyTorch network
A demand regressor accepts three input values—previous-week sales, discount fraction, and holiday timing. It chains torch.nn.Linear(input_size, 10), a sigmoid, torch.nn.Linear(10, 5), another sigmoid, and a final torch.nn.Linear(5, 1) that produces the prediction. Calling the model passes the input through this sequence. section 8.5
Stacking linear layers without nonlinear activations does not increase nonlinear expressive power: their composition is still equivalent to one linear layer. Activations such as tanh, ReLU, or sigmoid let a network model nonlinear combinations of its inputs. From a geometric view, successive layers can transform input manifolds until a final linear separator can distinguish classes. section 11.1 section 12.3
class TwoLayeredNN(torch.nn.Module):
def __init__(self, input_size, hidden1_size, hidden2_size, output_size):
super(TwoLayeredNN, self).__init__()
self.model = torch.nn.Sequential( #1
torch.nn.Linear(input_size, hidden1_size), #2
torch.nn.Sigmoid(),
torch.nn.Linear(hidden1_size, hidden2_size), #3
torch.nn.Sigmoid(),
torch.nn.Linear(hidden2_size, output_size) #4
)
def forward(self, X): #5
return self.model(X)
nn = TwoLayeredNN(input_size=X.shape[-1], hidden1_size=10,
hidden2_size=5, output_size=1)
Inspect a compact torch.nn.Sequential implementation with three linear layers, two sigmoid activations, and a forward method. section 8.5
