An inverted index makes full-text search fast by mapping each important word to the documents—and often the positions—where it appears. Like a textbook index, it lets a search system jump to matching documents instead of scanning every document one at a time. section 3.6.2 section 10.3.1
A practical indexing pipeline works like this:
- List the words in every document to create a forward index. section 10.3.1
- Invert that structure into word entries containing the documents where each word occurs. section 10.3.1
- Store word positions when searches need exact locations or phrase matching. section 3.6.2
- Use character trigrams to improve matching across spelling variations such as cat and cats. section 10.3.1
Searching a corpus with sparse vectors
A Python workflow can fit scikit-learn’s TfidfVectorizer to 41,531 Hacker Public Radio sentences, transform the corpus into a sparse matrix, transform “where is the lost audio” into a query vector, and select the highest-scoring document with a dot product. This illustrates retrieval over indexed word features before using a dedicated full-text engine. section 3.6.2 section 3.6.2
Listing 3.16 Downloading Hacker Public Radio show notes
>>> from sklearn.feature_extraction.text import TfidfVectorizer
>>> url = 'https://gitlab.com/tangibleai/community/knowt/-/raw/main/'
>>> url += '.knowt-data/corpus_hpr/sentences.csv?inline=false'
>>> df = pd.read_csv(url)
>>> docs = df['sentence']
>>> vectorizer = TfidfVectorizer(min_df=1) #1
>>> vectorizer = vectorizer.fit(docs) #2
>>> vectors = vectorizer.transform(docs) #3
<41531x56404 sparse matrix of type '<class 'numpy.float64'>'
with 788383 stored elements in Compressed Sparse Row format>
Inspect the runnable corpus-download and TF–IDF setup that produces the sparse document matrix used for search. section 3.6.2
Inverted indexes suit discrete or sparse word features because most document dimensions are zero. Dense, continuous semantic vectors from methods such as LSA and LDiA cannot be indexed as easily with traditional inverted-index techniques, so semantic search uses different retrieval approaches. section 4.8.1
