liveBook · Manning

Lexical search

As explained in AI-Powered Search

Keyword matching is a retrieval technique that identifies content by comparing query keywords with indexed textual terms.

Lexical search matches and ranks documents by the terms or attributes specified in a query. It is a practical starting point for a search engine: an explicit query can retrieve matching indexed content without a user profile or a model that understands the domain. Matching is only the first step; ranking determines which matches appear first. section 7.4.3 section 1.2.3 section 3.0

An inverted index maps each term in document fields to a postings list of documents containing it. Query terms provide fast lookups; Boolean operations can combine matching document sets, while positional matching can account for term sequences. One way to picture the process is as sparse query and document vectors with a dimension for each indexed term: present terms have a value of 1, absent terms 0. A relevance score ranks the matches; BM25 is one lexical ranking approach. section 2.1.3 section 2.3 section 7.1

A keyword match is not necessarily the right answer

For a reviews search, `bbq near charlotte` returned just one matching review: it contained all three words, but described a festival rather than a barbecue restaurant. Removing `near` produced more results, yet still missed relevant places whose reviews did not say `charlotte`. The example exposes two separate problems: a word can make a query too restrictive, and matching words need not capture intent. section 7.2

Listing 4.3 Running a search on the product catalog
def product_search_request(query, param_overrides={}):
  request = {"query": query,
             "query_fields": ["name", "manufacturer", "long_description"],
             "return_fields": ["upc", "name", "manufacturer",
                               "short_description", "score"],
             "limit": 5,
             "order_by": [("score", "desc"), ("upc", "asc")]}
  return request | param_overrides

query = "ipod"
request = product_search_request(query)
response = products_collection.search(**request)
display_product_search(query, response["docs"])

This Python search request selects product fields to query, returns a score, and sorts five matches by that score. It provides a small starting point for testing a lexical baseline. section 4.1.2

Exact terms are useful when the query names content directly, but they cannot by themselves establish meaning. A query for `soda` cannot retrieve related drinks if that term is absent from the index; `What is DEET?` can rank unrelated questions highly because their wording overlaps. Improve a baseline by testing real queries, tuning fields and analysis, or adding domain-aware query interpretation and expansion. Keep keyword matching as a fallback when richer interpretation cannot resolve a query. section 2.3 section 13.4.1 section 1.3 section 2.5.2

How AI-Powered Search covers it

How deep it goes
Explains inverted-index postings, sparse term vectors, and ranking at a high level; defers the specific scoring calculation.
Examples
Tests product-catalog searches, restaurant-review queries, and outdoor-question queries to expose useful matches and misleading keyword overlap.
What it assumes
Assumes familiarity with queries and documents while explaining inverted indexes and sparse term representations from their basic roles.
What you can do after
Apply it via an indexed collection and search request, then evaluate result relevance before adding richer matching.
How it connects
Connects inverted indexes and BM25 to semantic search, query expansion, personalized search, and dense vector search.

Also explained in

Compare how 8 books explain lexical search →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI-Powered Search. The section links point to the parts of the book it draws on.