A stop word is a frequent function word—such as “a,” “the,” “and,” or “of”—that usually contributes little substantive information to a phrase’s meaning. Filtering these tokens can reduce NLP computation, but their relationships may still carry essential meaning. section 2.3.1
Choose stop-word handling for the application rather than applying a universal deletion rule:
- Removing stop words can reduce the number of tokens an NLP pipeline must extract, store, and maintain. section 2.3.1
- Retaining stop words can preserve relational information inside n-grams, even when the individual words appear uninformative. section 2.3.1
- Deleting them can collapse distinct meanings: “reported to the CEO” and “reported as the CEO” both become “reported CEO.” section 2.3.1
- Document-wide frequency statistics can provide an alternative to relying entirely on an arbitrary stop-word list. section 3.1
Preserving professional hierarchy
“Mark reported to the CEO” suggests that Mark reported upward, while “Suzanne reported as the CEO to the board” identifies Suzanne as the CEO. Keeping stop words preserves this distinction in four-grams; removing them erases it. section 2.3.1
Listing 2.2 A broad list of stop words
>>> import requests
>>> url = ("https://gitlab.com/tangibleai/nlpia/-/raw/master/"
... "src/nlpia/data/stopword_lists.json")
>>> response = requests.get(url)
>>> stopwords = response.json()['exhaustive'] #1
>>> tokens = 'the words were just as I remembered them'.split() #2
>>> tokens_without_stopwords = [x for x in tokens if x not in stopwords]
>>> print(tokens_without_stopwords)
['I', 'remembered']
A Python example loads an exhaustive stop-word list, filters a tokenized sentence, and leaves “I” and “remembered.” section 2.3.1
