concept semantic search in category nlp

This is an excerpt from Manning's book Natural Language Processing in Action: Understanding, analyzing, and generating text with Python. Login to get full access to this book.

Over the next few chapters, we dive down through the top few layers of NLP. The top three layers are all that’s required to perform meaningful sentiment analysis and semantic search, and to build human-mimicking chatbots. In fact, it’s possible to build a useful and interesting chatbot using only a single layer of processing, using the text (character sequences) directly as the features for a language model. A chatbot that only does string matching and search is capable of participating in a reasonably convincing conversation, if given enough example statements and responses.

These topic vectors will help you do a lot of interesting things. They make it possible to search for documents based on their meaning—semantic search. Most of the time, semantic search returns search results that are much better than keyword search (TF-IDF search). Sometimes semantic search returns documents that are exactly what the user is searching for, even when they can’t think of the right words to put in the query.

This is called “semantic search,” not to be confused with the “semantic web.”^[52] Semantic search is what strong search engines do when they give you documents that don’t contain many of the words in your query, but are exactly what you were looking for. These advanced search engines use LSA topic vectors to tell the difference between a Python package in “The Cheese Shop” and a python in a Florida pet shop aquarium, while still recognizing its similarity to a “Ruby gem.”^[53]