A parameter is a value that controls a model’s computation and can be altered by an optimizer to reduce a loss. Learning consists of finding parameter values that improve predictions. section 1.4 section 1.7
For a model written as ŷ = fΘ(x), Θ denotes the complete parameter state. Gradient descent uses the loss ℓ(ŷ, y) and its gradients to update Θ during training. section 2.1.1
In PyTorch, parameter optimization follows a compact loop:
- Create a trainable value with nn.Parameter; it behaves like a tensor while identifying the value to adjust. section 1.4
- Pass the parameters to an optimizer such as SGD and choose a learning rate η. section 1.4
- For each epoch, clear gradients, compute the loss, call loss.backward(), and call optimizer.step() to update the parameters. section 1.4
A trainable scalar
To minimize f(x) = (x − 2)², represent x as x_param = torch.nn.Parameter(torch.tensor([-3.5]), requires_grad=True), then optimize it with torch.optim.SGD([x_param], lr=η). Repeated backward and step calls move the parameter toward the minimizing value. section 1.4
Model parameters commonly include weight matrices and bias terms. A linear layer uses W for the matrix-vector product and b to shift the output; nn.Linear(d, C) packages both as a module. section 2.1.5
