liveBook · Manning

LogSoftmax

As explained in Deep Learning with PyTorch, Second Edition

The logarithm of the softmax normalization used to produce log-probabilities for classification.

LogSoftmax converts classification scores into log-probabilities. It is useful before negative log likelihood because it computes logarithms stably when probabilities approach zero. section 7.2.4

Use LogSoftmax when:

  • A model predicts class scores for supervised classification. section 7.2.4
  • The next loss is nn.NLLLoss, which expects log-probabilities rather than ordinary probabilities. section 7.2.4
  • Numerical stability matters for very small predicted probabilities. section 7.2.4

Binary bird-versus-airplane classifier

A PyTorch model can end with nn.LogSoftmax(dim=1), then pass its output and a tensor of class indices to nn.NLLLoss(). Cross-entropy loss provides an alternative by combining these operations internally. section 7.2.4

# In[16]:
model = nn.Sequential(
             nn.Linear(3072, 512),
             nn.Tanh(),
             nn.Linear(512, 2),
             nn.LogSoftmax(dim=1))

Inspect the sequential classifier whose final layer is nn.LogSoftmax(dim=1). section 7.2.4

How Deep Learning with PyTorch, Second Edition covers it

How deep it goes
Explains the loss interface, numerical-stability rationale, and relationship to cross-entropy without developing the underlying mathematics formally.
Examples
Uses a PyTorch binary CIFAR bird-versus-airplane model and evaluates one image with NLLLoss.
What it assumes
Assumes supervised classification with class labels while introducing log-probabilities and their loss interface.
What you can do after
Apply it via PyTorch by ending a classifier with nn.LogSoftmax and pairing it with nn.NLLLoss.
How it connects
Connects softmax, negative log likelihood, NLLLoss, and cross-entropy loss.
Cover of Deep Learning with PyTorch, Second Edition
liveBookmanning.com
Read Deep Learning with PyTorch, Second Edition on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Deep Learning with PyTorch, Second Edition. The section links point to the parts of the book it draws on.