A weighted sum multiplies each value by a matching numerical weight, adds the results, and may include a bias. It lets a neuron combine inputs while giving some inputs more influence than others. section 9.20
In neural-network calculations, the operation follows a repeatable pattern:
- Multiply every input by the weight on its connection. section 9.4
- Add the weighted inputs for each neuron to produce its weighted sum. section 9.4
- Apply an activation function, such as sigmoid, to transform the sum into the neuron’s output. section 9.22
- Use a dot product for one neuron or matrix multiplication for many neurons at once. section 9.20 section 9.21
A neuron evaluating collision risk
For inputs [0.542, 0.5, 0.5, 0.2] and weights [3.35, -5.82, 0.85, -4.35], calculate (0.542 × 3.35) + (0.5 × -5.82) + (0.5 × 0.85) + (0.2 × -4.35). This single dot product replaces an explicit loop. In a layer with five hidden nodes, matrix multiplication performs the corresponding weighted sums together before activation functions produce node outputs. section 9.20 section 9.21
Figure 9.17 Weighted sum calculation for each hidden node
Shows the weighted-sum calculation performed separately for each hidden node. section 9.4
Table 11.21 A subset of tokens and their result values
|
q |
Weight |
V0 |
V1 |
V2 |
V3 |
V0 result |
V1 result |
V2 result |
V3 result |
| 0 |
0.0632 |
0.0150 |
1.9940 |
1.0050 |
1.0040 |
0.0009 |
0.1261 |
0.0635 |
0.0635 |
| 1 |
0.0090 |
0.8415 |
1.5543 |
0.5533 |
1.8424 |
0.0075 |
0.0139 |
0.0050 |
0.0165 |
| … |
|||||||||
| 31 |
0.4064 |
-0.0328 |
2.4097 |
0.2967 |
2.0802 |
-0.0133 |
0.9794 |
0.1206 |
0.8455 |
Shows how attention weights scale Value-vector components before the weighted results are added into a context vector. section 11.21
