Response streaming delivers an LLM answer incrementally rather than waiting for generation to finish. In a RAG chat interface, this gives users visible progress while the answer is being produced; retrieved sources can appear before the first answer chunk. section 12.3 section 12.4.4
A practical RAG streaming path is:
- Retrieve documents and prepare source information before generating the answer. section 12.3
- Yield a sources event, then iterate over `llm.astream(...)` and yield nonempty chunks as token events. The async generator gives the interface updates as they become available. section 12.3
- Connect the generator to a Chainlit message handler. Check initialization and validate the query before starting the response; keep RAG logic in a reusable core rather than embedding it in the UI. section 12.3
Listing 12.15 Streaming query processing with real-time updates
async def process_query_stream(self, #1
query: str) #1
-> AsyncGenerator[Dict[str, Any], None]: #1
try:
docs = self.retriever.invoke(query)
sources = [...] # Source preparation code
yield {
"type": "sources", (C)#2
"content": None,
"sources": sources,
"doc_count": len(docs)
}
context = "\n\n".join([...])
prompt = f"""..."""
async for chunk in self.llm.astream( #3
prompt,
config={"callbacks": [self.langfuse_handler]}
):
if chunk.content:
yield {
"type": "token", #4
"content": chunk.content,
"sources": sources,
"doc_count": len(docs)
}
yield {
"type": "done", #5
"content": None,
"sources": sources,
"doc_count": len(docs)
}
except Exception as e:
yield {
"type": "error", #6
"content": str(e),
"sources": [],
"doc_count": 0
}
The streaming query listing shows the event boundary: source metadata is yielded first, followed by chunks from the language model’s async stream. section 12.3
Measure the experience with both time-to-first-token and total generation time. A response that displays sources immediately can feel faster than a shorter wait with no feedback; one total-latency number misses that difference. Trace retrieval, context assembly, and generation when diagnosing delays or poor answers. section 12.4.4
