Softmax turns a set of raw scores into nonnegative probabilities that sum to one. In a neural-network classifier, the final layer produces one score, or logit, per class. Softmax makes those scores usable as a probability distribution for evaluating predictions. section 9.7
For each input, the forward pass:
- Computes the class logits from the network’s weights and biases. section 9.7
- Exponentiates the logits and divides each result by the sum across classes: `probs = exp_scores / np.sum(exp_scores, axis=1, keepdims=True)`. section 9.7
- Uses the predicted probability of the true class to calculate negative log-likelihood loss. section 9.7
Attention over graph neighbors
A graph attention network assigns raw attention scores to a node’s neighbors. Softmax normalizes those scores into comparable attention coefficients within the neighborhood. After multi-head attention, a final softmax may also turn classification outputs into probabilities. section 11.3.2
Choosing the next convex-hull point
A pointer network produces a softmax distribution over the points in its input sequence. At each step, that distribution assigns a probability to each candidate point, supporting the choice of which point to add next to the convex hull. The number of candidates follows the input length. section 11.9
def forward_prop(params):
logits = logits_function(params) #1
exp_scores = np.exp(logits) #2
probs = exp_scores / np.sum(exp_scores, axis=1, keepdims=True) #2
correct_logprobs = -np.log(probs[range(num_samples), y]) #3
loss = np.sum(correct_logprobs) / num_samples #4
return loss #4
Inspect the NumPy forward pass to see where logits become probabilities and where those probabilities enter the loss calculation. section 9.7
