liveBook · Manning

Knowledge boundary

As explained in Retrieval Augmented Generation, The Foundational Ideas

A self-knowledge boundary is the limit distinguishing information a language model can reliably know internally from information requiring external retrieval.

A manufacturing assistant may explain supply-chain principles yet miss yesterday’s port closure. Its answer draws on knowledge learned during training, while the delivery decision depends on a newer fact. The knowledge boundary is the point where the model’s internal world knowledge ends: its trained weights cannot contain the outcome of an event that happened after training stopped. section 1.3.1

Retrieval-Augmented Generation (RAG) bridges that boundary by fetching relevant external information for a query and giving it to the model as context for an answer. This differs from relying on the model’s parametric knowledge alone. It also keeps the freshness problem distinct from two related risks: a model can make an unsupported claim even about an older topic, and private organizational information may never have been available during public training. section 1.3.1 section 1.3

Use the boundary to decide what an answer needs before choosing a more complex pipeline:

  • Ask whether the fact changes faster than retraining can keep up, or is too rare to have been learned reliably. Either condition strengthens the case for retrieval. section 1.3.5
  • Retrieve the relevant passage rather than assuming a longer prompt should contain an entire collection. Extra context can obscure the fact the answer needs. section 1.3.4
  • Return supporting passages when a person must check the answer. Context can improve accuracy; a passage supplied with the answer also gives the reviewer something specific to verify. section 1.3.5

A ruling issued this morning

A legal assistant can analyze established precedents but cannot know a court ruling issued after its training cutoff from its weights alone. Fetching the new ruling gives it current context; returning the relevant passage lets a reviewer check whether the answer reflects that ruling. Retrieval addresses access to the new fact, not every possible reasoning or hallucination error. section 1.3.1 section 1.3.5

Start with basic retrieve-and-generate behavior when it meets the need. Improve retrieval and context processing when measured failures warrant it; add routing, retries, or agent-directed steps only for problems those simpler approaches do not solve. Fine-tuning, longer context windows, and caching may help specific tasks, but none removes the need to find external facts that the model cannot already know. section 1.3.4 section 1.6

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Defines the training cutoff and explains retrieval as a bridge, without specifying an implementation of boundary detection.
Examples
Uses a port closure, a new court ruling, and current clinical-trial results to show where trained knowledge falls short.
What it assumes
Introduces trained weights, parametric knowledge, and the training cutoff while assuming familiarity with language-model answers.
What you can do after
Evaluate/understand when changing or rare facts justify retrieval and when returned passages are needed for verification.
How it connects
Connects retrieval-augmented generation to parametric knowledge, hallucination, private knowledge, fine-tuning, and long-context prompting.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.