An SQL query selects the database records that an ingestion workflow will transform and index. In a product-data pipeline, the query is executed against products.db before records are summarized, cleaned, embedded, and uploaded to Azure AI Search. For example, SELECT * FROM Products supplies the product rows to process. section 3.2.6
Treat the query as the boundary between relational data and the search pipeline. A cursor executes the query, then the workflow fetches rows in batches rather than loading every matching record at once. Each record in a batch is passed to summarize_embed_upload(), while a running counter reports upload progress. section 3.2.6
For production-oriented ingestion, pair query execution with controlled batch handling:
- Connect to the database and create a cursor before executing the SQL query. section 3.2.6
- Fetch 100 rows at a time as an example batch size, and adjust that size for the environment. section 3.2.6
- Transform each fetched record through summarization, cleaning, embedding, and upload. section 3.2.6
- Stop when no records remain, then close the cursor and connection. section 3.2.6
- Track processed records so a failed or timed-out run can resume without re-embedding earlier uploads. section 3.2.6
Selecting products for indexing
Use ingest_all_records("SELECT * FROM Products") to execute a query against products.db. The resulting rows are fetched in groups of 100; each row is transformed and uploaded, and the workflow prints the cumulative upload total. After ingestion, compare that total with the search-index count and use Search Explorer to confirm records appear. section 3.2.6
Listing 3.8 Process and upload all records
def ingest_all_records(sql_query):
conn = sqlite3.connect("products.db") #A
cur = conn.cursor()
cur.execute(sql_query) #B
batch_size = 100 #C
total_records_transformed = 0 #D
while True:
records = cur.fetchmany(size=batch_size) #E
if not records:
break #F
for record in records:
print(f"Summarizing record with ID {record[0]}")
summarize_embed_upload(record)
total_records_transformed += len(records) #G
print(f"Total records uploaded: {total_records_transformed}")
cur.close()
conn.close() #H
if __name__ == "__main__":
ingest_all_records("SELECT * FROM Products") #I
The listing shows where the SQL query enters the ingestion function, how cur.execute(sql_query) runs it, and how fetchmany() turns its result set into bounded upload batches. section 3.2.6
Queries also appear at retrieval time in a different role: a user’s conversational question can be rewritten into a search query, which agents use to retrieve relevant embedded chunks for answer generation. This distinguishes database selection during ingestion from search querying during RAG inference. section 1.5 section 3.2.6






























