Information retrieval finds documents by searching their textual contents, especially keywords, instead of relying only on structured fields. It provides a practical way to locate reliable human-authored information for search tools and language-model applications. section 10.3 section 10.3.1
The central data structure is an inverted index. A forward index lists the words in each document; inversion produces a word-to-document lookup, much like a textbook index. Searching for cat can then jump directly to matching documents rather than scanning every document. section 10.3.1
A useful retrieval pipeline combines complementary methods:
- Use TF-IDF vectors for fast lexical matching across documents. section 10.3
- Use character-based trigram indexing to improve matching for variations such as cat and cats, and for typos or misspellings. section 10.3.1 section 3.9
- Combine text search with semantic search and an LLM when a question-answering system needs both speed and semantic recall. section 10.3
Finding cat and cats
A word-only index may fail when a query contains cat but a document contains cats. Character trigrams extend the matching surface so related spellings, and potentially misspellings, can be retrieved without reading every document sequentially. section 10.3.1 section 3.9
