Hyperparameter tuning selects training or generation settings that are not learned as model parameters. For large language models, changing data quality often delivers more value than searching these settings. section 5.4.4 section 7.2.2
Use tuning selectively and match the setting to the job:
- Use Optuna when a small metric improvement matters and the dataset cannot change soon. section 5.4.4
- Improve the dataset, represent the target metric more completely, or try oversampling before investing heavily in tuning. section 5.4.4
- For generation, adjust temperature and beam count only when the queried API endpoint accepts those parameters. section 7.2.2
- Prefer low or zero temperature for repeatable outputs; higher temperature increases response creativity but can reduce predictability. section 7.2.2
Temperature changes token selection
Given “I am a,” the model produces logits, converts them to probabilities with softmax, and applies temperature during that conversion. Zero temperature uses the highest-logit token for consistent output; higher values flatten the distribution. section 7.2.2
Figure 7.3 A simple path of how the next word is chosen. Given an input, a model will generate a vector of logits for each token in the model’s vocabulary. Using the softmax algorithm, these logits will be transformed into probabilities. These probabilities will correspond to how often that token is likely to be chosen. Temperature is applied during the softmax algorithm.
Follow the next-token path from logits through softmax probabilities to temperature-controlled selection. section 7.2.2
