liveBook · Manning

Torch.log_softmax

As explained in Build a Reasoning Model (From Scratch)

A PyTorch function that computes log-softmax over logits to produce log probabilities in a numerically stable way.

torch.log_softmax converts logits into log probabilities in one optimized step. It is used where token scores need to stay comparable while avoiding the extra work of chaining softmax and log. section 5.5 section 7.3.2

It preserves ranking: larger logits still become higher probabilities and less negative log probabilities. For scoring next tokens, that makes the best candidate the least negative log-probability. section 5.5 section 7.3.2

Used here to support probability-based scoring and training diagnostics:

  • computes per-token log probabilities from model logits section 5.5
  • supports next-token and sequence log-probability sums section 5.5
  • feeds entropy calculations over vocabulary distributions section 7.3.2

Next-token scoring

A prompt like “The capital of Germany is Berlin” is scored by taking model logits, applying torch.log_softmax, selecting the log probability for each observed next token, and summing them; an implausible token like “Bridge” produces a much lower total. section 5.5 section 7.3.2

How Build a Reasoning Model (From Scratch) covers it

How deep it goes
Shows the optimized torch.log_softmax form, its equivalence to softmax-plus-log, and how to gather token scores and sum them.
Examples
Uses a toy seven-token example, plotted logits/probabilities/log-probabilities, and prompt scoring for “Berlin” versus “Bridge”.
What it assumes
Assumes logits, softmax, and basic tensor indexing; it also teaches the logprob idea from the ground up.
What you can do after
Apply it to build stable token-scoring and entropy calculations from model logits.
How it connects
Connects logits, softmax, log probabilities, sequence log probability, and entropy.
Cover of Build a Reasoning Model (From Scratch)
liveBookmanning.com
Read Build a Reasoning Model (From Scratch) on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a Reasoning Model (From Scratch). The section links point to the parts of the book it draws on.