A parameter, often called a weight, is a number stored by a neural-network layer that helps determine how the layer transforms its input. Learning means finding values for the network’s parameters that map example inputs to their targets. A large network may have millions of them. section 1.6
A layer transformation
Consider `output <- relu(matmul(input, W) + b)`. Here `W` is a weight matrix and `b` is a bias; both are trainable parameters. When they start with random values, the output is unlikely to be useful. Training gradually changes them so the transformation better fits the data. section 2.4 section 2.4
A training loop gives those changes direction:
- Run a batch of inputs through the model and compare its predictions with the targets using a loss function. section 2.4
- Update the trainable parameters to reduce that loss, then repeat. The optimizer uses gradients to guide the updates. section 1.6 section 2.4
Not every stored weight is trained this way. In Keras, a Dense layer’s kernel and bias are trainable weights, while a BatchNormalization layer also keeps nontrainable weights for information about the data passing through it. When retrieving gradients for a custom training loop, use `model$trainable_weights`, not all `model$weights`. Parameters also differ from hyperparameters: the former are learned weights; the latter include choices such as the number and size of layers. section 7.4.1 section 5.2.1
Weights in a word-based model
A bag-of-words sentiment model can assign a learned weight to each word. A word such as “terrible” might point toward a negative review, while “riveting” might point toward a positive one. This gives a concrete picture of what a weight can contribute, though the model discards word order. section 14.4
