3 Search service ingestion

 

This chapter covers

  • Preparing structured data like SQL records for uploading to a search engine
  • Building the search engine and uploading the processed records
  • Building a hybrid search function to search the records by vector and keywords
  • Assessing which search services to use and which ones to avoid

Welcome to Chapter 3! Let’s talk about something that might sound a little controversial in today’s AI hype cycle:

You don’t have to use vectors and embeddings for RAG.

Blasphemy, you say. Hang him from the rafters, you shout angrily. Let me explain:

In RAG, your job isn’t to find similar vectors. It’s to feed the language model the right text. You may find that text however you please: if you have a 30-year-old SQL search that gives you the right answer every time, use that for retrieving the context for your RAG. If a quick Google search gives you all the information you need, then use the Google search API in your RAG pipeline. I encourage you to think outside the box: every single RAG tutorial I have ever seen on Youtube starts with “The first step in RAG is to build a vector database…” I shout at the screen: no, no, NO! Do not just assume that a vector database would be the best way for your RAG system. You might be choosing the wrong tool for the job, like trying to skin a fish with a hammer. How you search your data in your RAG system should be the best, fastest, and most accurate way for YOUR data, whatever form that takes on.

3.1 Setting up Azure AI Search

3.2 Preparing and uploading SQL records to AI Search

3.2.1 Code setup

3.2.2 Summarize one SQL record

3.2.3 Embed the record summary

3.2.4 Upload the record and embedding to Azure AI Search

3.2.5 Putting it all together

3.2.6 Loop through all SQL records

3.2.7 Test AI Search

3.2.8 Search filters and selecting fields

3.2.9 Ordering search results for consistency

3.2.10 Updating AI Search Index: adding new records

3.2.11 Updating AI Search Index: deleting records

3.3 Preparing and uploading unstructured data to AI Search

3.4 Summary