A context window is the bounded span of tokens an LLM can use when predicting its next token. In a chat application, the messages sent to the model and its generated response both count toward the model’s token limit. As a conversation grows, the application must manage that shared budget. section 2.8.6 section 3.4.4
Generation proceeds one token at a time: each predicted token joins the sequence used to predict the next. A longer window can accommodate more preceding information and help with long-range relationships, summarization, and other complex tasks. It is not automatically the better choice. Longer context increases computation and latency, so use a smaller window when it suffices. section 2.8.2 section 2.8.6
Budget a growing enterprise chat
Suppose a support chat sends a system message, prior questions and answers, and a new question. Each turn enlarges the message list. Before calling the API, count its tokens and leave room for the response; if the total will not fit, truncate, omit, or shorten content. An over-limit request can fail rather than produce an answer. Limiting response length does not remove the need to budget the input. section 3.4.4 section 6.7
Listing 3.25 ConsoleChatApp: Token management
import os
from openai import AzureOpenAI
import tiktoken
client = AzureOpenAI( #1
azure_endpoint=os.getenv("AOAI_ENDPOINT"), #1
api_version=”2024-05-01-preview”, #1
api_key=os.getenv(“AOAI_KEY”)) #1
GPT_MODEL = "gpt-35-turbo"
system_message = {"role": "system", "content": "You are
a helpful assistant #2
max_response_tokens = 250
token_limit = 4096
conversation = []
conversation.append(system_message) #2
def num_tokens_from_messages(messages): #3
encoding= tiktoken.get_encoding("cl100k_base") #4
num_tokens = 0
for message in messages: #5
num_tokens += 4
for key, value in message.items():
num_tokens += len(encoding.encode(value))
if key == "name":
num_tokens += -1
num_tokens += 2
print("I am a helpful assistant. I can talk about pets and salons.")
while True: #6
user_input = input("")
conversation.append({"role": "user", "content": user_input})
conv_history_tokens = num_tokens_from_messages(conversation)
while conv_history_tokens + max_response_tokens >= token_limit:
del conversation[1] #7
conv_history_tokens = num_tokens_from_messages(conversation)
response = client.chat.completions.create( #8
model=GPT_MODEL,
messages=conversation,
temperature=0.8,
max_tokens=max_response_tokens)
conversation.append({"role": "assistant", "content":
response.choices[0].message.content})
print("\n" + response.choices[0].message.content)
print("(Tokens used: " + str(response.usage.total_tokens) + ")")
The console chat listing uses tiktoken to count message tokens alongside a configured token limit and response allowance; treat the values as model-specific, not universal defaults. section 3.4.4
For questions over enterprise data, fitting more documents into one request is not the same as selecting relevant, current information. Retrieval-augmented generation (RAG) can bring enterprise data into the response instead of relying on window size alone. Keep prompts concise, test whether necessary details remain available, and account for token costs and the risk of incorrect answers. section 8.8 section 6.7 section 8.1.1
