4 Generation Pipeline: Generating Contextual LLM Responses
This chapter covers
- Retrievers and Retrieval Methodologies
- Augmentation with Prompt Engineering Techniques
- Generation using Large Language Models
- Basic implementation of the RAG pipeline in python
In Chapter 3, we discussed the creation of the knowledge base or the non-parametric memory of RAG-based applications via the Indexing Pipeline. To leverage this knowledge base for accurate and contextual responses a Generation Pipeline including the steps of retrieval, augmentation and generation is created.
In this chapter, we will elaborate the three components of the Generation Pipeline. We will begin by discussing the retrieval process which primarily involves searching through the embeddings stored in vector databases of the knowledge base and returning a list of documents that closely match the input query of the user. We will understand the concept of retrievers and a few retrieval algorithms. We will then move to the augmentation step. At this point, it will also be worthwhile understanding the different prompt engineering frameworks that are used with RAG. Finally, as part of the generation step, we will discuss a few stages of the LLM lifecycle like using foundation models vs supervised fine-tuning, models of different sizes and open-source vs proprietary models in the context of RAG. In each of these steps, we will also highlight the benefits and drawbacks of different methods.