liveBook · Manning

Structured data

As explained in Enterprise RAG

Structured data is information organized according to a predefined schema, such as rows and fields in database records.

Structured data organizes information into predefined fields, such as rows in SQL records. For retrieval, those records must be transformed into searchable text and vectors rather than uploaded as raw database rows. section 3.2 section 3.4

A production ingestion workflow processes structured records as follows:

  • Connect to the SQL database and fetch records in batches to avoid database and retrieval bottlenecks. section 3.2
  • Summarize each record into text, remove unnecessary or unusual characters, and generate an embedding. section 3.2 section 3.4
  • Package the text and vector as a search document, then upload it to Azure AI Search. section 3.2
  • Keep the index synchronized by updating modified records and removing deleted records. section 3.4
Figure 3.3 Workflow for ingesting structured SQL data into Azure AI Search. The process begins by connecting to an SQL database and fetching records in batches. Each batch is summarized, cleaned, embedded, and prepared as documents for upload. The pipeline loops through remaining records until all data is processed and uploaded.

The workflow diagram shows batch retrieval, record summarization, cleaning, embedding, document preparation, upload, and looping through remaining records. section 3.2

How Enterprise RAG covers it

How deep it goes
Traces ingestion from batched database queries through transformation, embedding, upload, and synchronization, without implementation code.
Examples
Uses batched SQL records that are summarized, cleaned, embedded, packaged, uploaded, updated, and deleted in the search index.
What it assumes
Assumes access to a SQL database, Azure AI Search, and an embedding API.
What you can do after
Apply it via Azure AI Search to build searchable representations of SQL records for hybrid retrieval.
How it connects
Connects SQL records with embeddings, vector search, text search, hybrid search, Azure AI Search, and enterprise RAG.

Also explained in

Compare how 31 books explain structured data →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Enterprise RAG. The section links point to the parts of the book it draws on.