A token limit caps how many tokens a model can accept as input. More room can provide useful context, but larger limits also increase memory demands, computation, latency, and failure risk. section 3.2.6
Treat token limits as a production constraint, not merely a model specification:
- Balance context against response speed: larger limits can require longer inference and produce more timeout errors. section 3.2.6
- Expect computational costs to rise sharply because attention calculations scale quadratically with sequence length. section 3.2.6
- Budget examples carefully in few-shot prompting; code samples can consume thousands of tokens and leave less space for the user. section 7.1.1
A larger limit can still be worse
After upgrading from GPT-3 to GPT-4, a team welcomed the higher token limit but encountered longer inference times and more timeout errors. A quicker, less accurate response may be preferable to no response. section 3.2.6
