BART is a denoising sequence-to-sequence Transformer that combines a bidirectional encoder with an autoregressive decoder. It reconstructs corrupted text during pretraining, learning to produce fluent, coherent language rather than merely extract spans. section 4.1.2
BART’s design supports generation from retrieved information:
- The encoder reads context bidirectionally, like BERT, while the decoder generates text token by token, like GPT. section 4.1.2
- A retrieved passage can be concatenated with a question and supplied as ordinary model input; changing the passage changes the generated answer without changing model weights. section 4.1.2
- In RAG, BART serves as the generator alongside a separately pretrained DPR retriever, forming a modular retrieve-then-generate pipeline. section 4.1
- Unlike extractive systems, BART can synthesize an answer whose wording does not appear verbatim in any one retrieved document. section 4.5
Retrieved question answering
A RAG system places a retrieved passage before the user’s question and feeds the combined text to BART. BART interprets the context and generates a fluent answer, even when the answer requires combining facts rather than copying one span. section 4.1.2 section 4.5
Figure 4.1 Compares the high-level architecture of REALM and canonical RAG, showing that RAG connects a pretrained retriever to a pretrained encoder-decoder generator while REALM jointly trains retrieval and the language model in a single, integrated system.

Compare BART-based RAG’s modular retriever-generator architecture with REALM’s jointly trained retrieval and language-model system. section 4.1


