concept document embedding in category deep learning

appears as: document embeddings, document embedding
Deep Learning for Search

This is an excerpt from Manning's book Deep Learning for Search.

You saw the effectiveness of document embeddings generated using word vectors in the small experiment with the “bernhard riemann influence” sample query. At the same time, in real life, you need better evidence for the effectiveness of a retrieval model. In this section, you’ll work on Similarity implementations based on averaged word2vec word vectors. You’ll then measure their effectiveness on a small dataset using the Lucene4IR project. These measures will give you a sense of how well these ranking models behave in general.

Extending a Lucene Similarity correctly is a difficult task that requires some insights into how Lucene works. We’ll focus on the relevant bits of the Similarity API to use document embeddings to score documents against queries. Let’s start by creating a WordEmbeddingsSimilarity that creates document embeddings via averaged word embeddings. It requires a trained word2vec model, a smoothing method to average word vectors to combine them in a document vector, and a Lucene field from which to fetch document content:

In the previous chapter, I introduced you to neural information retrieval models by building a ranking function based on averaged word embeddings. You averaged word embeddings generated by word2vec to obtain a document embedding, a dense representation of a sequence of words, that demonstrated high precision in ranking documents according to user intent.

The drawback of common retrieval models such as Vector Space Model with TF-IDF and BM25, however, is that they only look at single terms when ranking documents. This approach can lead to suboptimal results because the context information of those terms is discarded. With this drawback in mind, let’s see how you can generate document embeddings that look not just at single words, but at the whole text fragments surrounding those words. A vector representation created from these context-enhanced document embeddings will carry as much semantic information as possible, thus improving the ranking function’s precision even more.

Word embeddings are very good for capturing word semantics, but the meaning and deep semantics of text documents don’t depend on the meaning of words alone. It would be nice to be able to learn semantics about phrases or longer pieces of text instead of just words. In the previous chapter, you did that by averaging word embeddings. Going forward, you’ll discover that you can do better in terms of accuracy. In this chapter, we’ll explore a technique for learning document embeddings directly. Using extensions of the word2vec neural network learning algorithms, you can generate document embeddings for text sequences of different granularity (sentences, paragraphs, documents, and so on). You’ll experiment with this technique and show how it provides better numbers when used for ranking.

Additionally, you’ll learn how to use document embeddings to find related content. Related content consists of documents (texts, videos, and so on) that are semantically correlated. When you show a single search result (such as when the user clicks it in a search results web page), it’s common to display other content that, for example, deals with similar topics or was created by the same author. Doing so is useful to capture the user’s attention and provide them with content they might like but that may not appear on the first search results page.