Retrieval-augmented generation (RAG) lets a language model answer using relevant information retrieved from an external source. It is useful when answers depend on recent information or private enterprise data that the model did not learn during training. Rather than retraining the model whenever that data changes, an organization can update the external dataset. section 7.0 section 7.2
A RAG application has a retriever and a generator. The retriever searches an index, selects information relevant to the question, and supplies it as context. The generator—the language model—uses that context to produce an answer. In a vector-search implementation, the question and indexed documents are represented as vectors so the retriever can find similar documents. section 7.1 section 7.3
A practical implementation follows the data from source to answer:
- Inspect the source system and define the use case, users’ information needs, and success metrics before building the pipeline. section 8.4.1
- Extract and transform the data, split large documents into chunks, create embeddings, and store searchable records in an index. Choose chunk boundaries for the data and task; PDFs with tables or images need particular care. section 8.4.1 section 7.9
- At question time, search for relevant chunks, combine them with the question in a prompt, and call the language model. Keep the model’s token budget in view when selecting context. section 8.6
Chat with an archive of blog posts
A chat application chunks nearly 20 years of blog posts, embeds them, and indexes them in Redis. For each question, it retrieves the top five matches, adds them to the prompt, and calls a GPT model. Its prompt instructs the model to say when the articles do not contain an answer. This is an illustrative integration, not a production-ready deployment. section 8.6
Listing 8.9 End-to-end RAG-powered chat
def hybrid_search(query_vector, client, top_k=5, hybrid_fields="*"):
...
return results
def get_search_results(query:str, max_token=4096,
debug_message=False) -> str:
query_vector = get_embedding(query) #1
query_vector = np.array(query_vector).astype(
np.float32).tobytes() #2
print("Searching for similar posts...")
results = hybrid_search(query_vector, conn, top_k=5) #3
token_budget = max_token - count_tokens(query) #4
if debug_message:
print(f"Token budget: {token_budget}")
message = 'Use the blog post below to answer the subsequent
question. If the answer cannot be found in the
articles, write "Sorry, I could not find an answer in
the blog posts."'
question = f"\n\nQuestion: {query}"
if results:
for i, post in enumerate(results.docs): #5
next_post = f'\n\nBlog post:\n"""\n{post.content}\n"""'
new_token_usage = count_tokens(message + question + next_post)
if new_token_usage < token_budget:
if debug_message:
print(f"Token usage: {new_token_usage}")
message += next_post
else:
break
else:
print("No results found")
return message + question
def ask_gpt(query : str, max_token = 4096, debug_message = False) -> str:
message = get_search_results( #6
query,
max_token,
debug_message=debug_message)
messages = [ #7
{"role":
"system",
"content": "You answer questions in summary from the [CA]
blog posts."},
{"role":
"user",
"content": message},]
response = openai.ChatCompletion.create( #8
model="gpt-3.5-turbo-16k",
messages=messages,
temperature=0.7,
max_tokens=2000,
top_p=0.95
)
response_message = response["choices"][0]["message"]["content"]
return response_message
if __name__ == "__main__":
# Enter a query
while True:
query = input("Please enter your query: ")
print(ask_gpt(query, max_token=15000, debug_message=False))
print("=="*20)
The listing shows how a query embedding, top-five search, token budget, and retrieved context feed the chat request. section 8.6
Retrieval can ground responses and support source references, but it does not eliminate hallucinations. Answer quality depends on finding and preparing the right data. Production use also requires security, scaling, and operational work beyond a local Redis example. RAG complements prompt engineering and can remain useful even when a model is fine-tuned. section 7.2 section 13.1.2 section 8.6 section 9.0
