A shopper asks whether an item is available today. A stand-alone language model cannot know the store’s current inventory from its training data. Retrieval-augmented generation (RAG) addresses this gap: it fetches relevant information from an external source, then gives that information and the question to a language model to produce an answer. It retrieves and explains information; unlike an agent, RAG alone cannot place an order or change an external system. section 3.0 section 3.1 section 6.2.3
A practical pipeline has an indexing path and an answering path:
- Load source documents, split them into passages, embed the passages, and store them in a searchable index. Chunk size, embedding choice, and data freshness affect what the system can find. section 3.9
- Embed the question and retrieve relevant passages. Vector search can match meaning when wording differs; metadata filters and the number of returned results help narrow the context. section 1.4.1 section 4.5.4
- Pass the question and retrieved passages to the generator. Instruct it to answer from that context and return source information so the answer can be checked. section 4.5.4
A policy answer instead of a plausible guess
Asked about the return window for electronics bought with store credit, an ungrounded chatbot might guess “30 days.” A RAG retriever instead finds the store’s policy clause; the generator can answer with its specific 14-day store-credit refund window, 30-day exchange window, and receipt requirement. The detail comes from the retrieved policy, not a general retail rule. section 3.3.2
import openai
import os
openai.api_key = os.environ.get("OPENAI_API_KEY") #1
def answer_policy_question(query, filters=None):
"""
Answer healthcare policy questions using retrieved context
Args:
query: User question about healthcare policies
filters: Optional metadata filters
Returns:
Generated answer and sources
"""
results = search_policies(query, filters, top_k=2) #2
if not results:
return {
"answer": "I couldn't find specific information about that in our
policy documents.",
"sources": []
}
context = "" #3
sources = [] #3
#3
for i, result in enumerate(results): #3
context += f"\nPOLICY EXCERPT {i+1} #3
(Policy {result['policy_id']}):\n" #3
context += result['content'] #3
#3
source = { #3
"policy_id": result['policy_id'], #3
"section": result['section_title'], #3
"relevance_score": result['score'] #3
} #3
sources.append(source) #3
prompt = f""" #4
You are a healthcare policy assistant. Answer the question based ONLY
on the provided policy excerpts.
If the information is not in the excerpts, say you don't have enough
information.
POLICY EXCERPTS:
{context}
QUESTION: {query}
ANSWER:
"""
response = openai.chat.completions.create( #5
model="gpt-4.1",
messages=[
{"role": "system", "content": "You are a healthcare policy
assistant."},
{"role": "user", "content": prompt}
],
temperature=0.2,
max_completion_tokens=300
)
answer = response.choices[0].message.content.strip()
return {
"answer": answer,
"sources": sources
}
questions = [ #6
"Does physical therapy require prior authorization?", #6
"Do I need a referral to see a specialist?", #6
"Is my annual wellness visit covered at 100%?" #6
] #6
#6
for question in questions: #6
print("\n" + "="*50) #6
print(f"QUESTION: {question}") #6
response = answer_policy_question(question) #6
print(f"\nANSWER: {response['answer']}") #6
#6
print("\nSOURCES:") #6
for source in response['sources']: #6
print(f"- Policy {source['policy_id']}, " + #6
f"Section: {source['section']}, " + #6
f"Relevance: {source['relevance_score']:.2f}") #6
The health-care policy question-answering function shows the handoff: retrieve policy chunks, assemble context and source records, and return an answer with sources. It also supplies a response when retrieval finds nothing. section 4.5.4
Grounding reduces reliance on outdated model knowledge, but it does not make an answer automatically correct. Retrieval may miss a document; a passage may be stale or incomplete; the generator may still invent a detail. Keep sources visible, maintain the index, and evaluate retrieval precision, answer relevance, and factual accuracy. Try improved prompting first when the problem is only response behavior; use RAG when answers depend on changing facts, and consider fine-tuning for persistent style or format requirements. section 3.3.5 section 3.9 section 5.1.1
