A parameter is a trainable numerical value in a neural network. Training adjusts parameters to reduce prediction error and improve future accuracy; weights and biases are common examples. section 3.2.1 section 5.1.1
A training workflow changes parameters through a repeatable update loop:
- Feed a mini-batch through the network to produce predictions. section 3.2.1
- Compare predictions with labels through a loss function. section 3.2.1
- Use backpropagation to calculate gradients for each parameter. section 3.2.1
- Apply an optimization algorithm, such as stochastic gradient descent, using gradients and a learning rate. section 3.2.1
- Save the trained network and its parameters when the run or accuracy target is reached. section 3.2.1
Parameters versus hyperparameters
A model learns its parameters during training, while configurations such as 30 epochs, ReLU activation, minibatch size, and learning rate are set beforehand as hyperparameters. section 5.1.1
Use the workflow diagram to trace how predictions, loss, gradients, optimization, and saved parameters form one training cycle. section 3.2.1
