liveBook · Manning

Cross entropy loss

As explained in Build AI Drug Discovery Pipelines

Cross-entropy is a loss function that measures the discrepancy between a target probability distribution and a predicted probability distribution.

Cross entropy loss measures the discrepancy between target and predicted probabilities. It gives a training algorithm a numerical penalty for assigning low probability to the correct class; smaller loss means a better prediction. In drug discovery, it can train a classifier to identify compounds that inhibit a cytochrome P450 enzyme. section 5.1.1

For binary classification, the target y is 0 or 1 and the prediction ŷ is the probability of class 1. Binary cross entropy combines the penalties for both possible classes. For a true class of 1, a predicted probability of 0.3 produces a loss of about 1.20, while 0.8 produces about 0.22. Training aggregates losses across examples and uses gradient descent to update the model. section 5.1.1

LBCE(y,y^)=-(y*log⁡(y^)+(1-y)*log⁡(1-y^))

The binary cross-entropy equation shows how the target selects the penalty for the predicted class probability. section 5.1.1

For more than two mutually exclusive classes, softmax regression predicts a probability distribution across classes and minimizes multiclass cross entropy. This setup is unsuitable if a molecule can inhibit none or several of the enzymes: choosing one class forces exactly one answer. section 5.4.1

Classifying antibody sequences

A classifier can distinguish HIV-1 from SARS-CoV-2 antibody sequences using integer labels 0 and 1. PyTorch’s CrossEntropyLoss takes class logits and labels, applying softmax and negative log-likelihood internally in a numerically stable way. Evaluate the resulting classifier with metrics such as accuracy, precision, recall, F1 and area under the ROC curve, rather than relying on loss alone. section 12.3.4

loss_fn = nn.CrossEntropyLoss()
logits = model(batch)           # (B, num_classes)
loss = loss_fn(logits, batch["label"])  # scalar

This PyTorch example passes a batch of class logits and integer labels to CrossEntropyLoss. section 12.3.4

Reconstructing molecular strings

An autoencoder reconstructing SMILES strings predicts vocabulary scores at each sequence position. Cross entropy compares those scores with the original token indices. Set ignore_index=0 when zero represents padding, so padded positions do not contribute to the loss. section 10.2.3

Listing 10.5 Understanding the Loss Function
    def loss_function(self, recon, target, z=None):
        target = self._pad_or_truncate(target)    #A
        
        # Shape: (batch * max_length, vocab_size)
        recon_flat = recon.reshape(    #B
            -1, self.vocab_size    #B
        )    #B
        # Shape: (batch * max_length,)
        target_flat = target.reshape(-1)
        
        recon_loss = F.cross_entropy(    #C
            recon_flat,
            target_flat,
            ignore_index=0  # Don't compute loss on padding tokens
        )
        return {'loss': recon_loss, 'recon_loss': recon_loss}

The reconstruction loss reshapes predictions to (batch × sequence length, vocabulary size) and targets to a matching one-dimensional array before calling F.cross_entropy. section 10.2.3

How Build AI Drug Discovery Pipelines covers it

How deep it goes
Defines binary and multiclass loss, illustrates probability penalties, and implements loss calculations for sequence reconstruction and classification.
Examples
Illustrates inhibitor probabilities, antibody class labels and reconstruction of SMILES tokens with padding ignored.
What you can do after
Applies it via PyTorch CrossEntropyLoss and F.cross_entropy to class logits or token reconstruction scores.
How it connects
Connects cross entropy to logistic regression, softmax regression, gradient descent, classification and autoencoders.

Also explained in

Compare how 26 books explain cross entropy loss →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build AI Drug Discovery Pipelines. The section links point to the parts of the book it draws on.