A token limit constrains how much an AI model can process in one interaction. Input and generated output both matter: with an 8,192-token context window and a 200-token input, 7,992 tokens remain in that window. An oversized request can be rejected, so leave room for a useful response rather than filling the window with the prompt. section 5.1.1
Manage the budget at three points in a web application:
- Before sending: count prompt and system-message tokens, then compare their total with the model’s available context window. section 5.1.1
- When generating: set an output cap. An OpenAI chat completion uses max_tokens: 150; a streamText call can use maxTokens: 100. These settings cap the response, not the entire input-and-output window. section 2.5.5 section 8.2.2
- During ongoing conversations: remove older messages when history grows. Also check provider quotas such as tokens per minute or per day; those timeframe limits differ from a single interaction’s context window. section 8.2.2
Listing 5.1 Function that counts prompt tokens
import { encoding_for_model } from '@dqbd/tiktoken'; #1
function countTokens(text, model = "gpt-3.5-turbo") {
const enc = encoding_for_model(model);
const tokens = enc.encode(text); #2
enc.free();
return tokens.length;
}
const prompt =
"Translate the following English text to French: 'Hello, how are you?'";
const systemMessage
"You are a helpful assistant that translates English to French.";
const promptTokens = countTokens(prompt); #3
const systemTokens = countTokens(systemMessage); #4
const totalTokens = promptTokens + systemTokens;
console.log(`Prompt tokens: ${promptTokens}`);
console.log(`System message tokens: ${systemTokens}`);
console.log(`Total tokens: ${totalTokens}`);
JavaScript token-counting example: encode the prompt and system message with @dqbd/tiktoken, add their counts, and free the encoder. section 5.1.1
const response = await streamText({
model: supportedModel,
prompt: "Write a poem about love.",
maxTokens: 100, // Limit response to 100 tokens
});
A streamText request sets maxTokens to 100 to bound its generated response. section 8.2.2
