A noun phrase is a sequence centered on one or more nouns that represents a meaningful thing, person, place, or concept. In search, extracting noun phrases helps preserve meanings that keyword-by-keyword matching can lose. section 6.4.2 section 13.5.1
Why phrases matter in search
A query for “new york city” may be interpreted as separate Boolean terms—new, york, and city—so ranking scores each word independently. Recognizing the noun phrase preserves it as a meaningful search concept. section 6.4.2
A practical extraction pipeline can:
- Tokenize queries and remove syntax such as AND and OR before extracting phrases. section 6.4.2
- Use part-of-speech filtering or spaCy Matcher patterns to retain noun-tagged terms and reduce noise. section 6.4.2 section 13.5.1
- Count extracted concepts and keep frequent phrases for vocabulary or autocomplete suggestions. section 13.5.1
Listing 13.9 Using the spaCy Matcher to get desired parts of text
nlp = spacy.load("en_core_web_sm") #1
phrases = [] #2
sources = [] #3
matcher = Matcher(nlp.vocab) #4
nountags = ["NN", "NNP", "NNS", "NOUN"] #5
verbtags = ["VB", "VBD", "VBG", "VBN", ) #6
"VBP", "VBZ", "VERB"] #6
matcher.add("noun_phrases", [[{"TAG": {"IN": nountags}, #7
"IS_ALPHA": True, #7
"OP": "+"}]]) #7
matcher.add("verb_phrases", [[{"TAG": {"IN": verbtags},
"IS_ALPHA": True, "OP": "+",
"LEMMA":{"NOT_IN":["be"]}}]]) #8
for doc, _ in tqdm.tqdm(nlp.pipe(yield_tuple(dataframe, #9
source_field="body", #9
total=total),) #9
batch_size=40, #9
n_threads=4, #9
as_tuples=True), #9
total=total):
matches = matcher(doc)
for _, start, end in matches: #10
span = doc[start:end] #10
phrases.append(normalize(span)) #10
sources.append(span.text) #10
concepts = {}
labels = {}
for i, phrase in phrases: #11
if phrase not in concepts: #11
concepts[phrase] = 0 #11
labels[phrase] = sources[i] #11
concepts[phrase] += 1 #11
Use the spaCy Matcher listing as a concrete implementation pattern for selecting alphabetic noun sequences from processed documents. section 13.5.1
