A token limit caps how much an LLM can process or generate. Two limits matter in an API application: the model’s context-window limit and a configured response limit, often named `max_tokens`. The prompt and generated response both use tokens; depending on the model, they may share a maximum budget. A long prompt can therefore leave too little room for an answer. section 2.8.5 section 6.7
Manage the budget across the whole interaction:
- Set `max_tokens` to bound response length, but allow enough room for the requested answer. Generation can stop before a sentence is complete when it reaches the limit. section 2.8.5 section 3.4.4
- Count conversation tokens with `tiktoken`. If message history will not fit, truncate, omit, or shorten it, or limit the number of turns. section 3.4.4
- Watch token use as an operational cost: more tokens increase latency and expense. Keep prompts concise without removing information the task needs. section 3.4.4 section 6.7
Debugging a cut-off answer
A completions request sets `max_tokens=100` and asks for three outputs. One output returns `finish_reason` as `length`, indicating that generation stopped at the configured response limit or another token limit—not that the answer necessarily ended naturally. Check the finish reason before treating the output as complete; if appropriate, allow more response tokens while keeping the total interaction within the model’s limit. section 3.2.4 section 3.4.2
Listing 3.25 ConsoleChatApp: Token management
import os
from openai import AzureOpenAI
import tiktoken
client = AzureOpenAI( #1
azure_endpoint=os.getenv("AOAI_ENDPOINT"), #1
api_version=”2024-05-01-preview”, #1
api_key=os.getenv(“AOAI_KEY”)) #1
GPT_MODEL = "gpt-35-turbo"
system_message = {"role": "system", "content": "You are
a helpful assistant #2
max_response_tokens = 250
token_limit = 4096
conversation = []
conversation.append(system_message) #2
def num_tokens_from_messages(messages): #3
encoding= tiktoken.get_encoding("cl100k_base") #4
num_tokens = 0
for message in messages: #5
num_tokens += 4
for key, value in message.items():
num_tokens += len(encoding.encode(value))
if key == "name":
num_tokens += -1
num_tokens += 2
print("I am a helpful assistant. I can talk about pets and salons.")
while True: #6
user_input = input("")
conversation.append({"role": "user", "content": user_input})
conv_history_tokens = num_tokens_from_messages(conversation)
while conv_history_tokens + max_response_tokens >= token_limit:
del conversation[1] #7
conv_history_tokens = num_tokens_from_messages(conversation)
response = client.chat.completions.create( #8
model=GPT_MODEL,
messages=conversation,
temperature=0.8,
max_tokens=max_response_tokens)
conversation.append({"role": "assistant", "content":
response.choices[0].message.content})
print("\n" + response.choices[0].message.content)
print("(Tokens used: " + str(response.usage.total_tokens) + ")")
The token-management listing shows separate response and overall limits alongside `tiktoken`-based conversation counting. section 3.4.4
