A token limit bounds how much an LLM can process or generate in a request. The input limit constrains the prompt and retrieved context; the output limit constrains the response and is typically configurable with `max_tokens`. In a RAG application, both limits are design constraints: too much retrieved text can make a request fail, while an output cap can cut an answer off mid-sentence. section 12.1.1 section 12.3
Budget a retrieval request
Suppose retrieval supplies 50 chunks of 500 tokens each. That is 25,000 tokens before adding a 2,000-token system prompt and a 100-token query. Against a 32,000-token context window, the 27,100-token input leaves about 4,900 tokens for the response. More retrieved context is not automatically better if the answer cannot fit. section 12.1.1
Set the budget before sending a request:
- Choose how many chunks to retrieve and how long they should be; both determine how quickly context fills the input budget. section 12.1.1
- Set `max_tokens` to leave room for a complete answer without allowing unnecessarily long, costly generation. section 12.3
- Validate input before the LLM call so rejected requests do not consume the token budget. section 12.5
Listing 12.13 DakkaBotCore async initialization of RAG components
class DakkaBotCore: #1
def __init__(self):
self.retriever = None #2
self.llm = None
self.vector_store = None
self.initialized = False #3
async def initialize(self): #4
try:
embeddings = GeminiEmbeddings() #5
self.vector_store = FAISS.load_local(
"DATA/faiss_index",
embeddings=embeddings,
allow_dangerous_deserialization=True
)
self.retriever = self.vector_store.as_retriever(
search_kwargs={"k": 5} #6
)
self.llm = ChatGoogleGenerativeAI(
model="gemini-2.5-flash", #7
temperature=0.0,
google_api_key=os.environ["GEMINI_API_KEY"],
convert_system_message_to_human=True
)
self.initialized = True
except Exception as e:
raise RuntimeError(f"Failed to initialize DakkaBot: {str(e)}")
The retriever configuration sets `k` to 5. That retrieval count is one practical lever for controlling how much document text enters a RAG prompt; it is not itself an output-token setting. section 12.3
Streaming makes generated text visible as it arrives, but does not remove either limit. Budget for the full request even when the interface displays the answer token by token. section 12.3
