liveBook · Manning

Asynchronous index refresh

As explained in Retrieval Augmented Generation, The Foundational Ideas

An asynchronous index refresh is the process of rebuilding or updating a retrieval index during ongoing model training.

Asynchronous index refresh rebuilds a retrieval index while model training continues. It becomes necessary when document-encoder updates change document embeddings, making the existing index stale. section 4.4

The architecture separates ongoing training from periodic index rebuilding. It requires dedicated infrastructure, synchronization to preserve consistency, substantial compute, and custom distributed-training code. section 4.4

The main design tradeoff is whether encoder updates justify refresh complexity:

  • REALM updated the document encoder, query encoder, and generator together, forcing periodic refreshes of its document index. section 4.4
  • RAG avoided this cost by freezing the document encoder and fine-tuning only what was necessary, making training more practical for small teams. section 4.4

A costly refresh cycle

REALM indexed 13 million Wikipedia passages and reportedly required 400 TPU hours just to refresh its document index during training, making the approach impractical for many teams. section 4.4

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Traces encoder updates through periodic index rebuilding, infrastructure, synchronization, and TPU cost, but stops short of implementation detail.
Examples
Uses REALM's 13-million-passage Wikipedia index and 400 TPU hours for refreshes as the central comparison.
What it assumes
Assumes familiarity with retrieval-augmented generation and fine-tuning; defines the three jointly trained components by role.
What you can do after
Evaluate whether encoder updates justify asynchronous rebuilding when designing or comparing retrieval-augmented training.
How it connects
Connects retrieval indexes with document encoders, query encoders, generators, RAG, and REALM.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.