A neural network weight is a number that adjusts how strongly an input contributes to a result. Picture a volume control on each connection into a neuron: the input arrives, its weight changes its influence, and the neuron combines the weighted inputs before producing an output. The weights are learned parameters—they change as the network trains. section 9.2 section 9.8
From individual weights to a layer
Suppose a layer receives four inputs and has five hidden neurons. Each neuron needs a weight for each input, so the weights can be arranged in a 4 × 5 matrix. Multiplying the input vector by this matrix calculates all five weighted sums at once. A feed-forward layer uses the same idea to turn a four-value token vector into eight output values: each output comes from one column of weights. section 9.21 section 11.25
Table 11.27 Random weights for the project up layer
| 0.1 |
-0.2 |
0.3 |
0.4 |
-0.5 |
0.6 |
0.7 |
-0.8 |
| 0.9 |
0.8 |
-0.7 |
0.6 |
0.5 |
-0.4 |
0.3 |
0.2 |
| -0.3 |
0.2 |
0.1 |
-0.9 |
0.8 |
0.7 |
-0.6 |
0.5 |
| 0.5 |
-0.6 |
0.7 |
0.8 |
-0.9 |
0.1 |
0.2 |
-0.3 |
The grid of example weights makes the matrix concrete: each column supplies the weights used to calculate one new output value. section 11.25
A weight has two important moments in its life:
- Initialization gives it a starting value. Values that are too small or too large can make signals fade or grow erratically across layers; suitable initialization aims to keep signal strength reasonably steady. section 9.14 section 9.6
- Training adjusts it. The network compares a prediction with the actual value, then changes weights to improve later predictions. Unlike weights, the architecture chosen during setup stays fixed during this training process. section 9.8
A weight is not the same thing as every setting called a parameter. Population size in a genetic algorithm, for example, is a configured algorithmic parameter; a neural network weight is a number adjusted through learning. A bias is another learnable number: it shifts a node’s output rather than weighting an incoming value. section 4.20 section 9.8 section 11.25
The same idea scales beyond a single neuron. In a Transformer, parameters include trained weights; more weights give a language model capacity to represent finer relationships in its training data. Finding useful values can be computationally expensive, so particle swarm optimization is one possible search approach. For adapting a diffusion model, LoRA instead keeps its original weight matrices fixed and trains smaller matrices that add targeted changes. section 11.2 section 11.3 section 7.10 section 12.32
