When an app sends a prompt to a language model, the model reads it in tokens: chunks of text that may be whole words or parts of words. Its reply is made of output tokens. Both count when estimating API costs. section 9.6.2
A model builds a reply one token at a time, weighing possible next tokens. Temperature and top_p change how those possibilities are selected, affecting how predictable or varied the reply may be. section 9.6.1
Keep a trivia app affordable
A prompt that repeats every earlier question grows as the game continues. Keep prompts concise, ask for shorter replies, or generate several questions in one request. OpenAI’s tokenizer can help count tokens before you estimate costs. section 9.6.2
Tokens also measure a model’s context window: how much information can fit in a prompt. Sending an entire knowledge base with every question can be costly and may degrade results. Retrieval-augmented generation instead supplies relevant context. section 10.5.1
Cap a chatbot reply
Setting `max_tokens=2000` on `ChatOpenAI` limits a chatbot response to 2,000 tokens—about 1,500 words in the given example. This helps control response length and cost. section 10.2.3
Figure 9.10 How temperature and top_p affect the creativity and predictability of an LLM’s output.
Compare how temperature and top_p affect the predictability of the next token. section 9.6.1
