liveBook · Manning

Replace-on-reingest

As explained in Designing AI Systems

Replace-on-reingest updates indexed content by deleting data for an existing document and rebuilding it from the newly ingested version.

Replace-on-reingest keeps indexed content current by assigning a stable document identifier, deleting that document’s existing chunks and vectors, then extracting, chunking, embedding, and storing the new version. section 5.3.7 section 5.7

A production implementation should preserve these properties:

  • Use a caller-supplied stable document_id when available; otherwise derive an identifier from the filename. section 5.3.7
  • Treat deletion, extraction, chunking, embedding, and insertion as one atomic operation so failures restore the previous indexed version. section 5.3.7
  • Prefer full replacement when simplicity and correctness outweigh re-embedding unchanged sections; diff-based updates can fail when section boundaries shift. section 5.3.7

Updated support article

A revised support article is ingested with its existing document_id. The service removes its old chunks and vectors, processes the revised file, and inserts the replacement content without manual index cleanup. section 5.3.7 section 5.7

Listing 5.13 Document ingestion
def ingest_document(
    self,
    index_name: str,
    filename: str,
    file_bytes: bytes,
    metadata: Optional[Dict[str, str]] = None,
    document_id: Optional[str] = None    #A
) -> IngestedDocument:
 
    index = self.get_index(index_name)
    document_id = document_id or generate_document_id(filename)
 
    self.vector_store.delete_by_document(
        index_name, document_id
    )    #B
 
    extracted = self.pipeline.extract(filename, file_bytes)
 
    strategy = self.chunking_strategies[
        index.config.chunking_strategy
    ](
        chunk_size=index.config.chunk_size,
        chunk_overlap=index.config.chunk_overlap
    )    #C
    chunks = strategy.chunk(extracted)
 
    embeddings = self.embedding_generator.embed_chunks(
        chunks,
        model=index.config.embedding_model
    )    #D
 
    self.vector_store.insert(
        index_name, document_id, chunks, embeddings, metadata or {}
    )    #E
 
    return IngestedDocument(
        document_id=document_id,
        chunk_count=len(chunks),
        index_name=index_name
    )

Shows the ingestion service boundary and the ordered replacement pipeline, including document deletion, configurable chunking, embedding generation, and vector insertion. section 5.3.7

How Designing AI Systems covers it

How deep it goes
Defines deletion and full reprocessing, explains atomic failure handling, and stops short of implementing diff-based updates.
Examples
Uses revised policies, documentation, and support articles, plus a stable document_id and an ingestion service implementation.
What you can do after
Operate/configure document ingestion that replaces stale indexed content and preserves the prior version when processing fails.
How it connects
Connects document ingestion with chunking, embeddings, vector stores, stable identifiers, and database transactions.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Designing AI Systems. The section links point to the parts of the book it draws on.