Temperature sampling adjusts token probabilities before selection, controlling the trade-off between focused, predictable text and diverse, random text. section 4.1.5
The temperature modifies logits before softmax. Values below 1 sharpen probabilities around likely tokens; values above 1 flatten them and increase randomness. Temperature 1 leaves probabilities unchanged. section 4.1.5
Use temperature to tune generation behavior:
- Choose a low temperature for deterministic, focused output approaching greedy decoding. section 4.1.5
- Choose a high temperature for more varied output and exploration of less likely tokens. section 4.1.5
- Combine temperature with methods such as top-p to balance coherence and novelty. section 4.3
Transformers generation
Enable sampling in model.generate(), set temperature to 0.7, request 128 new tokens, and return three sequences. Changing the temperature changes the diversity and focus of the generated continuations. section 4.1.5
Listing 4.5 Temperature sampling implementation
temperature_outputs = model.generate( input_ids=input_ids, max_new_tokens=128, do_sample=True, temperature=0.7, num_return_sequences=3, pad_token_id=tokenizer.eos_token_id )
Inspect the executable temperature-sampling configuration, including sampling activation, temperature, sequence length, and multiple returned sequences. section 4.1.5
