A parameter is a value a model learns from data rather than a value you set as the final answer. In a simple temperature-calibration model, `t_c = w * t_u + b`, the weight `w` controls how strongly the input affects the prediction, while the bias `b` shifts it. Both are parameters: fitting the model means finding values that bring predicted temperatures close to measured ones. section 5.2.4
Start with two tensor parameters
With measured inputs `t_u` and targets `t_c` stored as tensors, the model can be written as: ```python def model(t_u, w, b): return w * t_u + b w = torch.ones(()) b = torch.zeros(()) t_p = model(t_u, w, b) loss = ((t_p - t_c) ** 2).mean() ``` Here `w` and `b` are scalar tensors. PyTorch broadcasts each across the input tensor; the mean squared difference gives one loss value to reduce. section 5.3.1
Training repeatedly makes predictions, compares them with targets, computes how the loss changes with each parameter, and updates the parameters in a direction that reduces loss. Think of `w` and `b` as two adjustment knobs. If updates are too large, training can overshoot and diverge; scaling the inputs can help when the weight and bias have very different gradient sizes. A learning rate controls the updates, but is a hyperparameter, not one of the values this model learns. section 5.2 section 5.4.3 section 5.4.4
As models grow, parameters take different shapes:
- A neuron can have a scalar weight and bias; a layer can have multidimensional weights and biases. A fixed activation function such as `tanh` has no trained parameters. section 6.1 section 6.3.2
- A convolution uses kernel weights across image neighborhoods. For `nn.Conv2d(3, 16, kernel_size=3)`, its weight tensor has shape `(16, 3, 3, 3)` and its bias has shape `(16,)`. section 8.1.1 section 8.2
Let a PyTorch module track them
`nn.Linear(1, 1)` owns `weight` and `bias` attributes. They are `nn.Parameter` tensors, which require gradients by default. Pass `linear_model.parameters()` to `optim.SGD` instead of gathering the tensors yourself. In a training step, clear old gradients, call `loss.backward()`, then call `optimizer.step()` to update them. For a model with several layers, `named_parameters()` also shows each parameter’s name and shape. section 6.2 section 6.2.2 section 6.3.2
# In[18]:
for name, param in seq_model.named_parameters():
print(name, param.shape)
# Out[18]:
0.weight torch.Size([13, 1])
0.bias torch.Size([13])
2.weight torch.Size([1, 13])
2.bias torch.Size([1])
The printed names and shapes show that each linear layer contributes its own weight and bias tensors. section 6.3.2
Once learned, weights and biases can be saved in a model’s `state_dict` and loaded into a newly created instance of the model. The saved parameter values do not, by themselves, include the model’s structure. Pretrained weights are the same idea at a larger scale: values already optimized on reference data can be loaded for a task instead of training from scratch. section 8.4.2 section 2.1
