A parametric-only model stores accessible knowledge entirely in learned parameters. Like an expert barred from consulting a library during an exam, it relies on facts distributed across its neural-network weights rather than retrieving external passages at query time. section 3.4
Rare facts expose the limitation
For the masked sentence “An equilateral triangle is easily constructed using a straightedge and compass, because 3 is a [MASK] prime,” standalone BERT has little chance of predicting “Fermat,” because the specific fact may not survive in its parameters. Adding a relevant passage through retrieval makes the answer substantially more probable. section 3.4
Figure 3.3 Prediction with and without retrieval, on REALM's own qualitative example.

Compare prediction from internal parametric knowledge with prediction after a retriever supplies supporting context. section 3.4
