liveBook · Manning

Indexing

As explained in LLM Evaluation and Alignment, The Foundational Ideas

Indexing is the process of transforming raw documents into a searchable representation.

Indexing turns raw documents into a searchable representation, preparing a corpus for sparse retrieval such as TF-IDF or BM25. It is analogous to cataloging books so readers can find them by title, topic, author, or subject. section 6.2.1

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Defines indexing as creating a searchable representation and compares it to library cataloging.
Examples
Uses library cataloging as an analogy for finding documents by title, topic, author, or subject.
What you can do after
Understand indexing as the corpus-preparation step required before sparse retrieval can work.
How it connects
Connects indexing to sparse retrieval, TF-IDF, BM25, and corpus processing.

Also explained in

Compare how 22 books explain indexing →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.