A training dataset is the portion of data used to fit a model. The optimizer adjusts parameters to reduce loss on these samples. Keep a separate validation set for checking how the model behaves on data it was not fitted to. section 5.5.3
# In[12]: n_samples = t_u.shape[0] n_val = int(0.2 * n_samples) shuffled_indices = torch.randperm(n_samples) train_indices = shuffled_indices[:-n_val] val_indices = shuffled_indices[-n_val:] train_indices, val_indices #1 # Out[12]: (tensor([9, 6, 5, 8, 4, 7, 0, 1, 3]), tensor([ 2, 10]))
For a small temperature-conversion example, `torch.randperm` shuffles sample indices. Tensor slicing assigns most indices to training and the remaining 20% to validation; the same indices must be applied to inputs and targets. section 5.5.3
Check both losses during training. Training loss shows whether the model can fit its training samples at all. If training loss falls while validation loss rises, the model is overfitting: it is learning details of the training set that do not generalize. Validation samples are evaluated, not used for parameter updates. section 5.5.3 section 14.5
# In[14]:
def training_loop(n_epochs, optimizer, params, train_t_u, val_t_u,
train_t_c, val_t_c):
for epoch in range(1, n_epochs + 1):
train_t_p = model(train_t_u, *params) #1
train_loss = loss_fn(train_t_p, train_t_c)
val_t_p = model(val_t_u, *params)
val_loss = loss_fn(val_t_p, val_t_c)
optimizer.zero_grad()
train_loss.backward() #2
optimizer.step()
if epoch <= 3 or epoch % 500 == 0:
print(f"Epoch {epoch}, Training loss {train_loss.item():.4f},"
f" Validation loss {val_loss.item():.4f}")
return params
# In[15]:
params = torch.tensor([1.0, 0.0], requires_grad=True)
learning_rate = 1e-2
optimizer = optim.SGD([params], lr=learning_rate)
training_loop(
n_epochs = 3000,
optimizer = optimizer,
params = params,
train_t_u = train_t_un, #3
val_t_u = val_t_un, #3
train_t_c = train_t_c,
val_t_c = val_t_c)
# Out[15]:
Epoch 1, Training loss 66.5811, Validation loss 142.3890
Epoch 2, Training loss 38.8626, Validation loss 64.0434
Epoch 3, Training loss 33.3475, Validation loss 39.4590
Epoch 500, Training loss 7.1454, Validation loss 9.1252
Epoch 1000, Training loss 3.5940, Validation loss 5.3110
Epoch 1500, Training loss 3.0942, Validation loss 4.1611
Epoch 2000, Training loss 3.0238, Validation loss 3.7693
Epoch 2500, Training loss 3.0139, Validation loss 3.6279
Epoch 3000, Training loss 3.0125, Validation loss 3.5756
tensor([ 5.1964, -16.7512], requires_grad=True)
The training loop calculates both losses, but calls `backward()` on training loss alone before the optimizer step. section 5.5.3
Training samples for next-character prediction
A dataset can require preparation before fitting. From the encoded name `$ada$`, make pairs such as input `$a` → target `d` and input `$ad` → target `a`. Multiple prefixes of one name become training samples. Pad sequences to a common length, ignore padded target values in the loss, and draw batches of input–target pairs. section 9.3
The split can also be built into a dataset: a CT-nodule dataset designates every tenth sample for validation and lets an option select training samples, validation samples, or all samples. After training and validation-based decisions are finished, use a separate test set for final evaluation. Avoid overlap or other information leakage between sets. section 12.5.2 section 5.5.4
