liveBook · Manning

SQL

As explained in Enterprise RAG

An SQL query is a statement written in SQL that requests or manipulates data in a relational database.

An SQL query selects the database records that an ingestion workflow will transform and index. In a product-data pipeline, the query is executed against products.db before records are summarized, cleaned, embedded, and uploaded to Azure AI Search. For example, SELECT * FROM Products supplies the product rows to process. section 3.2.6

Treat the query as the boundary between relational data and the search pipeline. A cursor executes the query, then the workflow fetches rows in batches rather than loading every matching record at once. Each record in a batch is passed to summarize_embed_upload(), while a running counter reports upload progress. section 3.2.6

For production-oriented ingestion, pair query execution with controlled batch handling:

  • Connect to the database and create a cursor before executing the SQL query. section 3.2.6
  • Fetch 100 rows at a time as an example batch size, and adjust that size for the environment. section 3.2.6
  • Transform each fetched record through summarization, cleaning, embedding, and upload. section 3.2.6
  • Stop when no records remain, then close the cursor and connection. section 3.2.6
  • Track processed records so a failed or timed-out run can resume without re-embedding earlier uploads. section 3.2.6

Selecting products for indexing

Use ingest_all_records("SELECT * FROM Products") to execute a query against products.db. The resulting rows are fetched in groups of 100; each row is transformed and uploaded, and the workflow prints the cumulative upload total. After ingestion, compare that total with the search-index count and use Search Explorer to confirm records appear. section 3.2.6

Listing 3.8 Process and upload all records
def ingest_all_records(sql_query):
  conn = sqlite3.connect("products.db")  #A
  cur = conn.cursor()
  cur.execute(sql_query)  #B
 
  batch_size = 100  #C
 
  total_records_transformed = 0  #D
  
  while True:
    records = cur.fetchmany(size=batch_size)  #E
    if not records:
      break  #F
 
      for record in records:
      print(f"Summarizing record with ID {record[0]}")
      summarize_embed_upload(record)
    
    total_records_transformed += len(records)  #G
    print(f"Total records uploaded: {total_records_transformed}")
 
  cur.close()
  conn.close()  #H
 
if __name__ == "__main__":
  ingest_all_records("SELECT * FROM Products")  #I
 

The listing shows where the SQL query enters the ingestion function, how cur.execute(sql_query) runs it, and how fetchmany() turns its result set into bounded upload batches. section 3.2.6

Queries also appear at retrieval time in a different role: a user’s conversational question can be rewritten into a search query, which agents use to retrieve relevant embedded chunks for answer generation. This distinguishes database selection during ingestion from search querying during RAG inference. section 1.5 section 3.2.6

How Enterprise RAG covers it

How deep it goes
Works through executing a SELECT query, fetching batches, tracking progress, and closing resources; it does not explain SQL syntax beyond the example.
Examples
Uses SELECT * FROM Products against products.db and verifies uploaded records in Azure Search Explorer.
What it assumes
Assumes a products.db database, a query string, and record-transformation functions; it demonstrates the execution loop directly.
What you can do after
Operate/configure it in a Python ingestion loop that processes query results in batches and resumes work after failures.
How it connects
Connects SQL queries to SQLite, cursors, batching, embeddings, Azure AI Search, and RAG retrieval.

Also explained in

Compare how 31 books explain SQL →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Enterprise RAG. The section links point to the parts of the book it draws on.