A corpus is a collection of documents assembled for linguistic analysis or information retrieval. Documents may combine analyzed text fields, while queries filter the corpus into a related document set. section 2.2
Search pipelines examine a corpus at several levels: characters, terms, term sequences, fields, documents, document sets, and the corpus as a whole. Text analyzers can split, lowercase, remove noise, stem or lemmatize, and strip accents before indexing. section 2.2
Use corpus-level evidence to interpret short queries and build search features:
- Look up query phrases in the corpus to discover common domain-specific interpretations. section 2.5.4
- Mine term co-occurrence across previous search sessions to infer intent that the query alone leaves implicit. section 2.5.4
- Extract frequent nouns and verbs from document text for autocomplete or nearest-neighbor vocabularies, filtering rare terms that add noise. section 13.5.1
Figure 2.8 Semantic data encoded into free text content. Characters form character sequences, which form terms, which form term sequences, which form fields, which form documents, which form a corpus.
Trace how characters become terms, fields, documents, and a corpus before search produces a query-specific document set. section 2.2
Filtering concepts from an outdoors corpus
A spaCy pipeline can process document bodies in parallel batches, match noun and verb phrases, exclude the verb “be,” and retain concepts occurring more than five times. This reduces autocomplete noise from misspellings and rare terms. section 13.5.1
