liveBook · Manning

Parametric-only model

As explained in Retrieval Augmented Generation, The Foundational Ideas

A parametric-only model is a model whose accessible knowledge is stored entirely in its learned parameters without consulting external information.

A parametric-only model stores accessible knowledge entirely in learned parameters. Like an expert barred from consulting a library during an exam, it relies on facts distributed across its neural-network weights rather than retrieving external passages at query time. section 3.4

Rare facts expose the limitation

For the masked sentence “An equilateral triangle is easily constructed using a straightedge and compass, because 3 is a [MASK] prime,” standalone BERT has little chance of predicting “Fermat,” because the specific fact may not survive in its parameters. Adding a relevant passage through retrieval makes the answer substantially more probable. section 3.4

Figure 3.3 Prediction with and without retrieval, on REALM's own qualitative example.

Compare prediction from internal parametric knowledge with prediction after a retriever supplies supporting context. section 3.4

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Defines parameter-stored knowledge and explains its failure on rare facts, but does not detail weight-level mechanisms.
Examples
Uses Fermat-prime prediction and Zeppelin-origin fill-mask outputs to compare unretrieved and retrieved knowledge.
What it assumes
Assumes familiarity with neural-network weights and BERT while explaining parametric memory from an intuitive analogy.
What you can do after
Evaluate when internal model memory is insufficient for specific facts and when retrieval can improve predictions.
How it connects
Connects parametric memory to BERT, retrieval, and retrieval-augmented generation.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.