A linear layer maps input features to output features through learned weights, producing one value for each output channel. In a fully connected layer, every output unit receives input from every unit in the preceding layer. section 3.1.2 section 3.3.3
Its practical role depends on where it appears. Attention uses separate linear layers to project embeddings into queries, keys, and values. A transformer feed-forward network expands the hidden representation, applies GELU, and contracts it back to the original size. section 3.3.3 section 3.3.4
Typical placements include:
- Attach a final linear layer to convert a model representation into class logits, such as 10 CIFAR-10 outputs. section 3.1.2
- Use three linear layers inside each attention head to create query, key, and value vectors. section 3.3.3
- Place linear layers before and after an activation in a position-wise feed-forward network. section 3.3.4
- End a convolutional image classifier with a fully connected layer that emits logits for the target classes. section 13.3.1
Transformer feed-forward network
With hidden_size 48 and intermediate_size 192, the first linear layer maps 48 inputs to 192 outputs; GELU transforms them; the second linear layer maps 192 outputs back to 48. section 3.3.4
class MLP(nn.Module):
def __init__(self, config):
super().__init__()
self.dense_1 = nn.Linear(config.hidden_size, #1
config.intermediate_size)
self.activation = GELU() #2
self.dense_2 = nn.Linear(config.intermediate_size,
config.hidden_size) #3
def forward(self, x):
x = self.dense_1(x)
x = self.activation(x)
x = self.dense_2(x)
return x
Inspect the MLP implementation and trace how two linear transformations surround GELU. section 3.3.4
