Mean absolute error (MAE), also called L1 loss, averages the absolute differences between corresponding predicted and target values. For images, that means comparing pixels. Unlike mean squared error (L2 loss), it does not square large differences, so it is less sensitive to outliers. section 6.3.2 section 15.1.3
The target depends on the task:
- In an autoencoder, compare a reconstructed handwritten digit with its original image. Minimizing MAE helps the decoder reproduce the input. section 7.0
- In a CycleGAN, use L1 loss for cycle consistency between images; use L2 loss for the adversarial loss. section 6.3.2
- In a denoising U-Net, compare predicted noise with the noise actually added to an image, then update the model to reduce that error. section 15.1.3
A pixelwise calculation
Suppose an original image and its generated counterpart each have shape (3, 256, 256). Take the absolute difference at each of their 196,608 corresponding positions, then average those differences to obtain the L1 loss. section 6.3.2
Listing 15.6 Training the denoising U-Net model
for epoch in range(num_epochs):
model.train()
tloss = 0
print(f"start epoch {epoch}")
for step, batch in enumerate(train_dataloader):
clean_images = batch["input"].to(device)*2-1
nums = clean_images.shape[0]
noise = torch.randn(clean_images.shape).to(device)
timesteps = torch.randint(0,
noise_scheduler.num_train_timesteps,
(nums, ),
device=device).long()
noisy_images = noise_scheduler.add_noise(clean_images,
noise, timesteps) #1
noise_pred = model(noisy_images,
timesteps)["sample"] #2
loss=torch.nn.functional.l1_loss(noise_pred, noise) #3
loss.backward()
optimizer.step() #4
lr_scheduler.step()
optimizer.zero_grad()
tloss += loss.detach().item()
if step%100==0:
print(f"step {step}, average loss {tloss/(step+1)}")
torch.save(model.state_dict(),'files/diffusion.pth')
In the denoising training loop, `torch.nn.functional.l1_loss(noise_pred, noise)` compares predicted noise with the sampled target noise. The resulting loss is backpropagated before the optimizer step. section 15.4.1
