Data ingestion turns source documents into searchable chunks. A shared pipeline lets teams ingest PDFs, Word documents, HTML, Markdown, and plain text without each building its own parsers. Custom parsers can handle unusual formats while reusing the same chunking, embedding, and storage infrastructure. section 5.3 section 5.3.1
An ingestion pipeline moves each document through distinct stages:
- Detect the format and route the file to a parser. Extract clean text while preserving headings, paragraphs, and section boundaries for structure-aware chunking. section 5.3.2
- Capture document-derived metadata and caller-supplied fields. Carry that metadata with each chunk so retrieval can filter results by attributes such as department or document type. section 5.3.2
- Apply the index’s chunking strategy, generate an embedding for each chunk through the Model Service, and store the vectors with their text and metadata. section 5.3.2 section 5.7
Figure 5.3 The document ingestion pipeline. A source file enters the pipeline and passes through five stages: (1) Format Detection identifies the file type (PDF, DOCX, HTML, Markdown, TXT). (2) Text Extraction uses format-specific parsers to produce clean text while preserving structural information like headings and paragraphs. (3) Metadata Extraction captures both automatic metadata (filename, page count, word count, ingestion timestamp) and user-supplied metadata (department, document type, author, tags). (4) Chunking splits the extracted text into semantically coherent pieces according to the index's configured strategy. (5) Embedding converts each chunk into a dense vector through the Model Service. The output is a set of vectors with their original text and metadata, ready for storage in the vector store.

Follow the stages from source file to vectors ready for storage. section 5.3.2
Index creation and ingestion are separate operations. An index starts as an empty container with embedding and chunking configuration; ingestion adds documents and can run again when sources change. Track the last ingestion time to check whether scheduled synchronization is current. section 5.2.3
Chunk boundaries affect retrieval quality: splitting critical information across chunks can prevent retrieval of a complete answer. Preserving document structure helps a structure-aware strategy choose useful boundaries. section 5.7 section 5.3.2
Safe re-ingestion
A stable document ID lets an updated document replace its previous chunks. Deleting old chunks before inserting replacements can leave a gap if embedding or storage fails. Make the replacement atomic so failure restores the previous state. For a bulk parser update that must not expose a mixed index, ingest into a new index, verify it, and then swap traffic. section 5.3.7
For large documents or batches, accept requests asynchronously and return a job ID immediately. A caller can poll for queued, processing, completed, or failed status; progress and error fields make long-running work observable and support retries after failure. section 5.3.9
Listing 5.15 Asynchronous ingestion interface
@dataclass
class IngestJob:
job_id: str #A
status: str #B
document_id: Optional[str] = None #C
progress: float = 0.0 #D
error: Optional[str] = None #E
class DataService:
# ... other operations ...
def ingest(
self,
index_name: str,
filename: str,
file_bytes: bytes,
metadata: Optional[Dict[str, str]] = None
) -> IngestJob: #F
"""Accept a document for ingestion and return immediately."""
pass
def get_ingest_status(
self, job_id: str
) -> IngestJob: #G
"""Check the status of an ingestion job."""
passInspect the job fields and submit-and-poll interface for asynchronous ingestion. section 5.3.9








