Stop-word removal filters selected high-frequency function words from tokenized text, often before building bag-of-words vectors. The shortcut can shrink representations, but dropping articles, prepositions, or verb forms may discard meaning and reduce precision or accuracy. section 2.3.1 section 1.5.3
A practical stop-word filter involves:
- Tokenize the input before filtering; the filter operates on individual tokens. section 1.5.3
- Choose a maintained list from sources such as NLTK, spaCy, or broader multilingual lists rather than assuming every common word is disposable. section 2.3.1
- Inspect the filtered output and measure its effect, because exhaustive lists can remove important tokens and rarely guarantee better pipeline accuracy. section 2.3.1
Filtering a short sentence
Given the tokens “the words were just as I remembered them,” a broad list leaves only “I” and “remembered.” The result is smaller, but much of the original meaning has disappeared. section 2.3.1
