liveBook · Manning

Corpus

As explained in Retrieval Augmented Generation, The Foundational Ideas

A corpus is a collection of documents available as a body of text for retrieval or analysis.

A corpus is the document collection that supplies text for retrieval or analysis. In retrieval-augmented generation, it may contain PDFs, office documents, or web pages and acts as the knowledge source retrieved before generation. section 4.7.1

A practical RAG corpus typically connects these components:

  • Documents provide the searchable content. section 4.7.1
  • An embedding model represents corpus content for retrieval. section 4.7.1
  • A vector store such as Pinecone, FAISS, or Chroma holds searchable representations. section 4.7.1
  • A language model uses retrieved material to generate an answer. section 4.7.1

Corpus growth as a retrieval failure

When a corpus grows without corresponding updates to chunk size or retrieval depth, new documents can dilute context relevance and pull queries away from the right answers. Logging the full retrieved context helps distinguish retrieval failure from generation failure. section 15.6.1

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Defines corpus contents and pipeline role, then connects growth to retrieval dilution, chunk size, retrieval depth, and monitoring.
Examples
Uses PDFs, office documents, web pages, corporate documents, academic papers, and web content as corpus examples.
What you can do after
Diagnose retrieval versus generation failures by logging retrieved context and monitoring RAGAS and RAGGED trends.
How it connects
Links document corpora to retrieval-augmented generation, embedding models, vector stores, language models, chunking, and evaluation metrics.

Also explained in

Compare how 6 books explain corpus →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.