A corpus is a managed collection of documents or passages that a retriever searches for information relevant to a query. In retrieval-augmented generation (RAG), retrieved passages are provided to a generator model so it can produce more specific, current, and reliable responses. section 7.4 section 8.8
Select retrieval behavior according to query characteristics, domain needs, and available resources:
- Use sparse retrieval with techniques such as TF-IDF and inverted indexes for fast, keyword-dependent search at scale; it may miss synonyms and nuanced phrasing. section 7.4
- Use dense retrieval to encode queries and documents as vectors, capturing semantic relationships and context; it generally requires more computation and training data. section 7.4
Production corpora require quality control, regular updates, security, privacy protection, access control, testing, documentation, and resource planning. Enterprise teams can align retrieved information with internal policies and industry regulations while retaining control over proprietary data. section 8.4.1 section 8.8
Enterprise RAG corpus
An enterprise can assemble proprietary documents and diverse information sources into a retrieval corpus. RAG then retrieves relevant, up-to-date passages for time-sensitive questions and can provide supporting sources for generated content. section 8.8
Figure 7.3 Custom data on LLMs
Use the custom-data diagram to relate an enterprise corpus to the larger LLM and RAG architecture. section 7.4
