Softmax turns a set of scores into nonnegative weights that sum to one. In attention, those weights determine how much each input contributes to an output. In text generation, softmax probabilities determine how likely tokens are to be sampled. section 2.2.3 section 4.1.5
For scaled dot-product attention, first compare a query with the keys to produce scores. Apply softmax across those scores, then use the resulting weights to combine the values. In the implementation below, `axis=1` applies softmax across each row of query–key scores; multiplying by `V` produces the attention output. section 2.2.3
Listing 2.3 Simplified scaled dot-attention implementation
embed_1 = np.array([0, 1, 0]) #1 embed_2 = np.array([1, 0, 1]) embed_3 = np.array([0, 1, 1]) embed_4 = np.array([1, 1, 0]) embeddings = np.array([embed_1, embed_2, embed_3, embed_4]) #2 Wq = rand(3, 3) #3 Wk = rand(3, 3) Wv = rand(3, 3) Q = embeddings.dot(Wq) #4 K = embeddings.dot(Wk) V = embeddings.dot(Wv) attention_scores = softmax(Q.dot(K.T) / sqrt(K.shape[1]), axis=1) #5 attention_output = attention_scores.dot(V) #6
The `attention_scores` line shows softmax applied after scaling the query–key dot products; the next line uses those weights to combine values. section 2.2.3
Temperature changes the distribution *before* softmax is applied to token logits. A temperature of 1 leaves the probabilities unchanged. Higher temperatures make them more uniform and sampling more varied; lower temperatures concentrate probability on the likeliest tokens, approaching greedy decoding as temperature nears zero. section 4.1.5
Listing 4.5 Temperature sampling implementation
temperature_outputs = model.generate( input_ids=input_ids, max_new_tokens=128, do_sample=True, temperature=0.7, num_return_sequences=3, pad_token_id=tokenizer.eos_token_id )
This `model.generate()` call enables sampling and sets `temperature=0.7`, showing how to control the token distribution without implementing softmax directly. section 4.1.5
