Part 4 The search frontier

The rise of embeddings and generative AI has been a boon for the field of information retrieval. Not only do large language models (LLMs) and other foundation models provide new ways to understand and generate text, but they also serve as a perfect complement for search engines. Generative models need reliable data as context (which search engines provide), and search engines need to interpret and summarize the data they search (which generative AI models provide).

In part 4, we’ll explore the frontier of search. We’ll look at how generative models are being used to improve search, and how search is being used to augment generative models. We’ll also look at the emerging future at the intersection of AI and information retrieval.

Chapter 13 covers semantic search over embeddings, explaining how Transformers work and how semantic search over dense vectors can be optimized for efficiency with approximate nearest neighbor (ANN) and quantization approaches. Chapter 14 demonstrates how to fine-tune an LLM on your data and implement extractive question answering: responding to questions in queries with explicit answers extracted from search results.