Batching updates model parameters from a selected group of training examples rather than one example at a time. Each batch produces its own estimate of the error surface, so training can progress without treating any single sample as a complete guide. section 5.3.2 section 5.2.5
Training uses several batch granularities:
- Individual-example training calculates a gradient from one sample at a time and can wander because each sample provides an incomplete signal. section 5.3.2
- Full batch learning uses the entire training dataset for an update, while batch learning more generally uses a subset such as 0.1%, 1%, 10%, or 20% of the data. section 5.3.2
- Mini-batch learning balances noisy individual gradients with larger, more directed updates; batch sizes around 16 to 64 are described as effective for many neural-network problems. section 5.3.2
A PyTorch training loop
A neural network trains by clearing old gradients, running a forward pass on training data, calculating loss, backpropagating the error, and applying the optimizer step. The same forward pass can be organized around batches instead of individual samples. section 5.2.5 section 8.3
>>> from tqdm import tqdm >>> num_epochs = 200 >>> pbar_epochs = tqdm(range(num_epochs), desc='Epoch:', total=num_epochs, ascii=' =') >>> for epoch in pbar_epochs: ... optimizer.zero_grad() #1 ... outputs = model(X_train) ... loss_train = loss_func_train(outputs, y_train) #2 ... loss_train.backward() #3 ... optimizer.step() #4 Epoch:: 100%|================================| 200/200 [00:04<00:00, 42.84it/s] 96.26it/s]
Inspect the executable PyTorch loop that repeats optimization across epochs. section 5.2.5
Batching is not only a speed choice: changing the selected examples changes the estimated error surface. Learning rate and momentum still control how aggressively the optimizer responds to those estimates. section 5.3.2 section 5.2.5
