Batch gradient descent updates model parameters after computing the gradient over every available training example. Each update uses the full dataset rather than a randomly selected mini-batch, making the update more informed but more expensive. section 3.3.2 section 2.4.3
A two-class linear classifier
A pure TensorFlow example creates 2,000 synthetic two-dimensional samples, assigns binary targets, and trains a linear classifier. Each training step computes predictions and mean squared error for all samples, obtains gradients with GradientTape, and subtracts the scaled gradients from W and b. With a learning rate of 0.1, 40 steps reduce the loss from 4.5364 to 0.0289; predictions then separate the classes with a threshold of 0.5. section 3.3.2
A batch-gradient-descent training step follows this pattern:
- Run the forward pass on the entire training set. section 3.3.2
- Compute one loss value from the complete set of per-sample losses. section 3.3.2
- Differentiate the loss with respect to every trainable parameter. section 3.3.2
- Update the parameters using the learning rate, then repeat. section 3.3.2
Compared with mini-batch training, each batch update takes longer because it processes all samples at once. It can reduce training loss more effectively per step, require fewer steps, and support a larger learning rate; its main limitation is computational expense. section 3.3.2 section 2.4.3
learning_rate <- 0.1
training_step <- tf_function( #1
jit_compile = TRUE,
\(inputs, targets, W, b) {
with(tf$GradientTape() %as% tape, {
predictions <- model(inputs, W, b) #2
loss <- mean_squared_error(targets, predictions)
})
grad_loss_wrt <- tape$gradient(loss, list(W = W, b = b)) #3
W$assign_sub(grad_loss_wrt$W * learning_rate) #4
b$assign_sub(grad_loss_wrt$b * learning_rate) #4
loss #4
}
)
Inspect the complete R TensorFlow training-step listing: it combines the forward pass, loss calculation, automatic differentiation, and full-dataset parameter updates. section 3.3.2
