15 Foundation models and emerging search paradigms

 

This chapter covers

  • Leveraging foundation models for results summarization and abstractive question answering
  • Emerging search paradigms like multimodal search and conversational search
  • Leveraging AI for generative search and agent-based search pipelines
  • The future of AI-powered search

Large Language Models, as we’ve tested and fine-tuned in the last two chapters, are front and center in advances in AI-powered search over recent years. Between improving query interpretation and document understanding by mapping content into embeddings for dense vector search, to helping with answer extraction, you’ve already seen some of the key ways search quality can be enhanced by these models.

But what additional advanced approaches are emerging and on the horizon? In this chapter, we’ll cover some of the more recent advances at the intersection of search and AI. We’ll cover how Foundation models are being used to extend additional capabilities to AI-powered search like results summarization, multi-modal search, and even chatbot-enabled interfaces for search and information retrieval.

We’ll cover the basics of emerging search paradigms like generative search and new classes of foundation models that are reinventing some of the ways we’ll soon approach the frontier of AI-powered search.

15.1 Understanding foundation models

15.1.1 Training vs. fine-tuning vs. prompting

15.1.2 Evaluation

15.2.1 Results summarization using foundation models

15.2.2 Data generation using foundation models

15.2.3 Evaluating generative output

15.2.4 Constructing your own metric

15.3 Other emerging AI-powered search paradigms

15.4 Convergence of contextual technologies

15.5 Hybrid search: all the above, please!

15.6 Summary