liveBook · Manning

Cross entropy loss

As explained in Fight Fraud with Machine Learning

Cross-entropy loss measures the discrepancy between a target distribution and predicted probabilities, commonly using the negative log probability assigned to the target.

Cross-entropy loss measures classification error from predicted probabilities and ground-truth labels. It is defined as the negative logarithm of the prediction assigned to the correct class, so less confident correct predictions produce larger losses. section 6.4.3 section 9.2.3

Use cross-entropy loss to:

  • Train segmentation models that classify each receipt pixel as forged or genuine. section 6.3.2
  • Account for class imbalance by assigning larger weights to mistakes on rare forged pixels. section 6.3.2
  • Compare training and validation behavior, while interpreting larger losses cautiously when a task has more classes. section 6.4.3
  • Train classifiers distinguishing illicit from licit Bitcoin transactions before evaluating accuracy and F1 score. section 9.2.3

Weighted receipt segmentation

For a two-class receipt segmentation model, class frequencies such as [1.2, 1] are inverted to create weights, then passed to PyTorch's CrossEntropyLoss. Raising the forged-class weight makes forged-pixel errors contribute more; making it excessively large can push the model toward predicting every pixel as forged. section 6.3.2

class_frequencies = [1.2, 1]  #A
weights = 1.0 / torch.tensor(class_frequencies, dtype=torch.float32)  #B
optimizer = torch.optim.Adam(model.parameters(), lr=1e-4)  #C
loss_function = torch.nn.CrossEntropyLoss(weight=weights)  #C

Inspect the concise PyTorch setup for inverse-frequency class weights, the Adam optimizer, and weighted CrossEntropyLoss. section 6.3.2

def train():
      model.train()
      optimizer.zero_grad()  #A
      out = model(data.x)  #B
      loss = criterion(out[data.train_mask], data.y[data.train_mask])  #C
      loss.backward()  #D
      optimizer.step()  #E
      return loss

Follow the training step from forward pass through loss calculation, backpropagation, and optimizer update. section 9.2.3

How Fight Fraud with Machine Learning covers it

How deep it goes
Explains the negative-log relationship and weighted implementation, but defers a fuller mathematical derivation.
Examples
Uses forged receipt pixels, two- and three-class segmentation, and licit-versus-illicit Bitcoin transactions.
What it assumes
Assumes familiarity with classification, labels, model predictions, optimization, and training-validation splits.
What you can do after
Apply it via PyTorch to train classifiers, adjust class weights, and monitor training and validation loss.
How it connects
Connects class imbalance, classification, backpropagation, Adam optimization, validation sets, accuracy, and F1 score.

Also explained in

Compare how 26 books explain cross entropy loss →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Fight Fraud with Machine Learning. The section links point to the parts of the book it draws on.