A jointly trained retriever and reader can still underperform if fine-tuning leaves their capacity unused. REALM++ improves REALM without changing its architecture. It addresses undertraining and reports an accuracy gain of roughly 5.5 percentage points on a question-answering task. section 3.4
The changes focus on training and evaluation rather than a new retriever or reader:
- Increase the effective fine-tuning batch size. More diverse question–document pairings and hard negatives in each update help the retriever distinguish useful passages. section 3.4
- Use exact maximum inner product search (MIPS), instead of approximate nearest-neighbor search, for final evaluation. This checks performance when approximate retrieval does not affect which documents are selected. section 3.4
- Strengthen supervision alongside the scaling changes. The reported gain is for the improved approach as a whole, not an isolated estimate for this lever. section 3.4
The practical lesson is to test whether fine-tuning and retrieval evaluation are limiting an existing system before replacing its architecture. Exact MIPS is described for final evaluation; that choice alone does not establish a production retrieval strategy. section 3.4
