A linear layer transforms an input vector by multiplying it with learned weights and usually adding a bias. The weights reshape information; the bias shifts each output up or down, giving the network more flexibility. section 11.25
In a Transformer feed-forward block, a projection layer expands or transforms normalized token representations so later operations can learn richer features. Because these operations are linear, a nonlinear activation such as ReLU is applied afterward. section 11.25
From a token vector to a prediction
A final linear layer, called the language model head, multiplies the last Transformer output by a weight matrix whose second dimension equals the vocabulary size. Its outputs are logits—raw, unnormalized scores for possible next tokens. Softmax converts those scores into probabilities; choosing the largest score gives a predicted token. section 11.29
Table 11.27 Random weights for the project up layer
| 0.1 |
-0.2 |
0.3 |
0.4 |
-0.5 |
0.6 |
0.7 |
-0.8 |
| 0.9 |
0.8 |
-0.7 |
0.6 |
0.5 |
-0.4 |
0.3 |
0.2 |
| -0.3 |
0.2 |
0.1 |
-0.9 |
0.8 |
0.7 |
-0.6 |
0.5 |
| 0.5 |
-0.6 |
0.7 |
0.8 |
-0.9 |
0.1 |
0.2 |
-0.3 |
Inspect the sample weights used to illustrate how a project-up transformation combines input values into output dimensions. section 11.25
