When a retriever learns alongside a language model, its document embeddings change after gradient updates. A search index built from earlier embeddings quickly becomes stale, but rebuilding an index over millions of documents after every step is impractical. Asynchronous Maximum Inner Product Search (MIPS) refresh keeps training moving while the index catches up. section 3.2
Two jobs run in parallel:
- The trainer continuously updates the model using a slightly stale search index. section 3.2
- An index builder periodically recomputes all document embeddings and supplies a fresh index to the trainer. section 3.2
A 500-step refresh
The index builder might refresh the index every 500 training steps rather than after every update. Training proceeds in the meantime. This trades temporary index staleness for a tractable way to jointly pretrain the retriever and language model at scale; it does not make that pretraining inexpensive. section 3.2 section 3.5
