A hyperparameter is a configuration value set before model training that controls the learning process. Unlike model parameters such as weights and bias, it remains static during training and cannot be learned from data. section 5.0 section 5.1.1
Common hyperparameters include:
- Learning rate, which influences training behavior and speed. section 5.1.1 section 5.1.2
- Batch or minibatch size, which controls how training data is processed. section 5.0 section 5.1.1
- Epoch count, optimizer choice, activation function, layer count, neuron count, embedding dimensions, and regularization method. section 5.1.1 section 5.1.2
Hyperparameter values affect model accuracy, training speed, memory requirements, and learning behavior. Because many values may need testing, hyperparameter optimization automates the search for configurations that produce strong training results. section 5.0 section 5.5
Spiral classification in TensorFlow Playground
A configuration with six hidden layers, five neurons per layer, ReLU activation, batch size 10, and L1 regularization can classify a spiral-shaped pattern accurately after nearly 500 epochs. section 5.1.2
