liveBook · Manning

Stop word

As explained in Just Use Postgres!

A stop word is a frequent function word, such as an article or preposition, that is excluded during text normalization because it contributes little search meaning.

A stop word is a frequent function word, such as “the,” “and,” or “are,” removed during text normalization because it contributes little search meaning. Postgres removes stop words while converting tokens into lexemes, alongside case normalization and stemming. section 6.2.1 section 6.1

Removal does not erase positional information. In the sentence “The explorers must save the fragile peace between Earth and the aliens,” “the” and “and” disappear, while “explorers” becomes “explor” at position 2. Preserved positions support phrase and distance operators. section 6.2.1

Observe stop-word removal with to_tsvector

SELECT * FROM to_tsvector('The explorers must save the fragile peace between Earth and the aliens.'); Postgres returns lexemes such as 'explor':2, 'must':3, 'save':4, 'fragil':6, 'peac':7, 'earth':9, and 'alien':12. The articles and prepositions are absent, but their positions still affect phrase matching. section 6.2.1

The same normalization applies to search queries. Functions that produce tsquery values remove stop words before combining remaining lexemes with operators. Phrase queries account for omitted words by using distance operators when needed. section 6.3 section 6.3.2

How Just Use Postgres! covers it

How deep it goes
Explains removal during normalization and preserved positions, but defers detailed dictionary and configuration behavior.
Examples
Uses movie descriptions, to_tsvector output, and the phrase “The Lion King” to demonstrate omission and positional effects.
What it assumes
Assumes familiarity with PostgreSQL full-text search while defining tokenization, normalization, and lexemes along the way.
What you can do after
Apply stop-word-aware normalization with PostgreSQL’s to_tsvector and query-conversion functions.
How it connects
Connects tokenization, normalization, stemming, lexemes, tsvector, tsquery, phrase search, and distance operators.

Also explained in

Compare how 4 books explain stop word →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Just Use Postgres!. The section links point to the parts of the book it draws on.