liveBook · Manning

Stale data

As explained in Retrieval Augmented Generation, The Foundational Ideas

Staleness is the condition in which stored or indexed information no longer reflects the current underlying source.

A retrieved answer can be relevant, supported, and obsolete. Suppose an older policy permits an action, but a newer policy forbids it. If retrieval returns only the older document, an answer citing it may pass relevance and support checks while giving the wrong current rule. Staleness is therefore a question of whether the evidence still reflects the underlying source, not just whether it matches the query. section 9.5.1

Staleness appears at three different points:

  • Model knowledge: facts held only in model weights may be unavailable when they arise after training. An external document repository can be updated without retraining the whole model. section 3.4
  • Vector index: training a document encoder changes the embeddings it produces, leaving a precomputed index aligned with older encoder weights. section 6.4
  • Retrieved documents: an indexed passage may accurately describe an earlier state yet have been superseded by a newer source. section 9.5.1

The remedies depend on which layer is stale. Atlas recomputed its index every 1,000 pre-training steps, accepting roughly 30% overhead relative to training with a fixed retriever; re-indexing a large corpus after every few steps was considered infeasible. For outdated documents, use effective dates to filter or downweight expired material, link superseded documents to their replacements, and retire old records. At query time, a retrieval evaluator can flag possibly stale evidence and fall back to live web search when cached evidence is insufficient. section 6.2 section 6.4 section 9.5.1 section 11.1.3

A citation is not a freshness check

A query asks whether an entity is currently sanctioned. Retrieval finds a 2021 document saying the entity is under review but misses a 2024 document confirming delisting. Before answering, the system needs to assess whether the retrieved date fits the query’s implied recency and seek further evidence if it does not. section 11.1.3

More answer critique alone cannot repair a missing newer source: repeated checks of the same answer against the same old passage can keep approving it. Freshness controls must also govern what enters retrieval and when the system searches again. section 9.5.1 section 11.6

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Distinguishes stale model knowledge, embeddings, and documents, then gives refresh schedules and freshness controls without specifying a complete deployment procedure.
Examples
Uses an obsolete policy, a sanctions-status query, and Atlas index recomputation to show distinct failure modes.
What it assumes
Assumes familiarity with retrieval and document embeddings while explaining why encoder updates invalidate precomputed vectors.
What you can do after
Diagnose/debug with it by separating stale indexes from outdated documents and choosing an index refresh or a fresh-evidence check.
How it connects
Connects staleness to document embeddings, retriever training, relevance checks, supersession, retrieval evaluation, and live web search.

Also explained in

Compare how 3 books explain stale data →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.