A keyword search index organizes terms by their presence or absence in documents, making later retrieval efficient. A bag-of-words vector treats a document like a chord: its words appear together rather than replaying sequentially. section 2.5.2
A practical index-building workflow:
- Build a vocabulary from the unique tokens in the document collection. section 2.5.2
- Ignore uninformative stop words when they are unlikely to help keyword searches. section 2.5.2
- Encode each document as a fixed-length binary vector and use it to retrieve matching documents. section 2.5.2
Scaling the representation
Limiting a vocabulary to 10,000 important words compresses an imaginary 3,500-sentence book to about 10 kilobytes, while a 3,000-book corpus takes about 30 megabytes. The same representation supports fast document lookup and underlies tools such as Elasticsearch, Solr, and PostgreSQL. section 2.5.2
>>> bow = sorted(set(re.findall(pattern, text))) >>> bow[:9] [',', '.', 'Liesel', 'Trust', 'and', 'arrived', 'clouds', 'hands', 'her'] >>> bow[9:19] ['hold', 'in', 'like', 'me', 'on', 'out', 'rain', 'she', 'the', 'their'] >>> bow[19:27] ['them', 'they', 'though', 'way', 'were', 'when', 'words', 'would']
Inspect the extracted vocabulary and see the unique tokens that define the vector dimensions. section 2.5.2
