liveBook · Manning

Corpus

As explained in LLM Customization and Fine-Tuning

A body of text or other training data used to train a model or, here, to build the retrieval index for RAG.

A corpus is the collection of text chunks encoded to build a retrieval index. If you replace the sentence-transformer encoder, re-encode every chunk: embeddings from different models are not directly comparable. section 4.5.3

Choose the index layer to match corpus scale:

  • Use a flat NumPy array for a prototype with tens of thousands of chunks; consider FAISS for hundreds of thousands or a vector database for multi-tenant production with millions. section 4.5.3

How LLM Customization and Fine-Tuning covers it

How deep it goes
Explains re-encoding chunks after encoder changes, but does not specify corpus preparation or chunking procedures.
Examples
Uses a NumPy flat index for tens of thousands of chunks and recommends other index layers at larger scales.
What you can do after
Operate retrieval indexing by re-encoding every corpus chunk when replacing the sentence-transformer encoder.
How it connects
Connects corpus chunks to sentence-transformer encoders, vector indexes, and retrieval-augmented generation.

Also explained in

Compare how 6 books explain corpus →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.