Chapter 2: Vector similarity search and hybrid search

Introduction

This chapter introduces using embeddings, vector similarity search, and hybrid search in a RAG application to answer questions from unstructured data.

• Unstructured-data RAG workflow: Describe the chapter’s RAG workflow for answering questions over unstructured data: retrieve relevant information with search and use that information to generate an answer. Explain that knowledge-graph construction can begin iteratively with unstructured data and gain structure later.
  Why it matters: This establishes how a RAG application can provide useful answers before all source data has been fully structured into a knowledge graph.
• Embedding-based similarity search: Explain that embedding models transform complex data such as text and images into uniform representations called embeddings, enabling vector similarity search to find relevant information for RAG.
  Why it matters: Embeddings and similarity search provide the retrieval approach that the chapter uses to locate useful context in unstructured data.
• Hybrid search enhancement: State that the chapter adds full-text search to vector similarity search to form a hybrid search approach, with the aim of improving retrieval results.
  Why it matters: Hybrid search is the chapter’s next retrieval improvement beyond vector similarity search alone.

Key terms: RAG, unstructured data, knowledge graph, retriever, generator, embedding models, embeddings, vector space, vector similarity search, full-text search

2.1 Components of a RAG architecture

This section introduces the retriever and generator as the two main components of a RAG application and states their respective roles.

• Retriever and generator roles: Describe the two main components of a RAG application and explain that the retriever finds relevant information while the generator uses that information to create a response.
  Why it matters: Separating retrieval from generation clarifies how a RAG system grounds an LLM response in relevant information.
• Vector search in retrieval: State that vector similarity search is used by the retriever to find relevant information in a RAG application.
  Why it matters: This connects the retriever's job to the vector-search approach developed in the following sections.

Key terms: RAG application, retriever, generator, vector similarity search

2.1.1 The retriever

This section explains how a RAG retriever prepares and searches embedded text chunks to supply relevant context to a generator.

• Retriever role and pipeline: Describe the retriever’s role in a RAG application and trace its basic vector-retrieval pipeline from a user query to source chunks passed to the generator.
  Why it matters: The retriever determines which information the generator receives as grounding context.
• Vector index speed tradeoff: Explain what a vector index does and distinguish approximate nearest neighbor search with an index from a brute-force search in terms of speed and accuracy.
  Why it matters: This tradeoff explains why practical retrievers commonly favor indexed search over exact exhaustive comparison.
• Embedding representation consistency: Explain that an embedding model converts complex input such as text into numeric vectors that capture meaning or context, and state why one embedding model must be used consistently unless the vector index is repopulated.
  Why it matters: Vector matching works only when stored content and incoming queries are represented compatibly.
• Text chunking choices: Describe text chunking as splitting source text into smaller pieces for retrieval, explain why narrower chunks can improve relevance, and identify the chunking choices that depend on the use case, data, and domain.
  Why it matters: How content is chunked strongly affects whether retrieval returns focused, useful context.
• Cosine similarity selection: State how cosine similarity compares two vectors and why this section selects it for text chatbots instead of Euclidean distance.
  Why it matters: The similarity function determines how the retriever decides which embedded text is relevant to a query.

Key terms: retriever, generator, retriever pipeline, query, source chunks, vector similarity search, vector index, approximate nearest neighbor search, brute-force search, nearest neighbors

2.1.2 The generator

This section explains how the RAG generator uses retrieved information to produce answers and why this allows smaller, faster, cheaper, and more grounded language models.

• Generator role in RAG: Describe the generator as the second RAG component that uses information found by the retriever to generate a response.
  Why it matters: The generator turns retrieved context into the answer the user receives.
• Reduced generator knowledge burden: Explain why RAG does not require the generator to know everything: the retriever supplies relevant information, while the generator only needs to use that information to construct a response.
  Why it matters: Separating retrieval from response generation makes the language model's job substantially smaller.
• Smaller grounded language models: Explain that RAG uses the language model primarily for text generation rather than its stored knowledge, enabling smaller, faster, and cheaper models whose responses can be grounded in retrieved information and hallucinate less.
  Why it matters: This is a practical reason to use RAG when cost, speed, and factual grounding matter.

Key terms: generator, retriever, RAG application, response, base knowledge, RAG, language model, hallucinate, retrieved information, finetuning

2.2 RAG using vector similarity search

This section introduces the planned vector-similarity RAG application, its retriever-to-response flow, and its separation into data setup and query-time stages.

• Vector RAG application flow: Describe the intended flow of the RAG application: a retriever finds information through vector similarity search, and that retrieved information is used to generate a response.
  Why it matters: This establishes how retrieval contributes grounded information to the response-generation process.
• Data setup and query time: Distinguish the two stages of the application: data setup is considered first, followed by the work the application performs at query time.
  Why it matters: Separating preparation from request-time work makes the RAG pipeline easier to understand and implement.

Key terms: RAG application, vector similarity search, retriever, response, data setup, query time

2.2.1 Application data setup

This section introduces the data-preparation components and embedding flow that make vector similarity search possible in a RAG application.

• Vector search setup components: Describe the four pieces required for application data setup for vector similarity search: a text corpus, a text-chunking function, an embedding model, and a database with vector similarity search capability.
  Why it matters: These components prepare source text so the RAG application can retrieve relevant information at run time.
• Chunk embedding retrieval flow: Trace how text chunks are stored and retrieved: split corpus text is stored as chunks, embeddings of those chunks populate the vector index, and a user question is embedded with the same model so the index can find similar chunks.
  Why it matters: Using the same embedding model for documents and questions puts them in a comparable vector space for similarity retrieval.

Key terms: text corpus, text-chunking function, embedding model, database with vector similarity search ability, text chunks, vector index, embeddings, vector similarity search, vector space

2.2.2 The text corpus

This subsection introduces the Einstein patent paper as a corpus for demonstrating how RAG handles specific questions better than an LLM alone.

• Einstein patent paper corpus: Identify “Einstein’s Patents and Inventions” by Caudhuri (2017) as the text corpus used for the example.
  Why it matters: The corpus is the source against which the RAG application's answers are tested.
• Specific-question RAG comparison: Explain that the example demonstrates RAG by asking very specific questions and comparing answers grounded in the paper with answers from an LLM alone, despite the LLM already knowing about Einstein.
  Why it matters: This comparison makes the value of retrieval concrete when broadly known subjects require precise source-based answers.

Key terms: text corpus, Einstein’s Patents and Inventions, Caudhuri, 2017, RAG, LLM, specific questions, paper

2.2.3 Text chunking

This section explains how to split corpus text into overlapping, whitespace-aligned chunks to improve vector retrieval quality.

• Chunk size selection: Explain why a paper is split into smaller text chunks for retrieval even when an LLM context window could hold the whole paper, and state that the best chunk size must be determined experimentally for a particular case.
  Why it matters: Chunk boundaries and size affect what the retriever can find and therefore the quality of the generated answer.
• Overlapping sliding windows: Describe the sliding-window chunking approach used here: 500-character chunks with 40-character overlap. Explain that overlap helps retrieve answers spanning chunk boundaries, at the cost of a larger index, and improves retriever accuracy.
  Why it matters: Overlap preserves nearby context that would otherwise be separated at a chunk boundary.
• Whitespace-aligned chunk boundaries: Describe why the chunking function can split only at spaces: it prevents broken words at the beginnings and ends of chunks and helps the embedding model classify each chunk's semantics. State the function's inputs and output at a high level.
  Why it matters: Keeping words intact gives embeddings cleaner text to represent semantically.

Key terms: text chunking, chunk size, context window, overlap, sliding window, 500 characters, 40 characters, index, retriever, embedding model

2.2.4 Embedding model

This section selects a text embedding model for chunk embeddings and explains why the same model must be used consistently with the vector index.

• Text embedding model selection: Describe choosing an embedding model based on the kind of data to be matched, using a text-embedding model when matching text chunks. State that the example uses OpenAI embedding models, while all-MiniLM-L12-v2 through Sentence Transformers is presented as an alternative that can run on a local CPU.
  Why it matters: The embedding model turns text into the vectors that make semantic retrieval possible.
• Embedding-index model consistency: Explain why a RAG application must use the same embedding model throughout: the vector index contains vectors generated by that model. State that changing the embedding model requires repopulating the vector index.
  Why it matters: Vectors from a changed model cannot simply be mixed with the vectors already stored in the index.

Key terms: embedding model, text-embedding model, OpenAI, all-MiniLM-L12-v2, Sentence Transformers, Hugging Face, vector index, vectors, repopulate the vector index

2.2.5 Database with vector similarity search function

This section stores text-chunk embeddings in Neo4j by defining a simple Chunk model, creating a dimension-matched vector index, and loading the chunk data.

• Chunk node data model: Describe the Neo4j data model used for the initial vector-search RAG application, including the Chunk node and its stored properties.
  Why it matters: The retriever needs each embedding to remain linked to the text chunk it represents.
• Vector index configuration: Explain how the Neo4j vector index is configured: it indexes the embedding property of Chunk nodes with cosine similarity and must use the embedding model's vector dimension.
  Why it matters: A vector index can search embeddings efficiently only when its configuration matches the vectors being stored.
• Chunk loading and inspection: Describe how chunk data is populated and checked in Neo4j using Cypher: create a node for each chunk, set its text and embedding properties, retain a chunk index, and query a Chunk node such as the one with index 0.
  Why it matters: Loading and inspecting the nodes confirms that the content needed for later retrieval is present in the database.

Key terms: Chunk, text, embedding, vector index, pdf, cosine similarity, dimensions, Cypher, index

2.2.6 Performing vector search

This section embeds a user question and uses Cypher to retrieve and inspect the two most similar text chunks from the populated Neo4j vector index.

• Question embedding consistency: Explain why the user’s question is embedded with the same embedding model and embedding function used for the text chunks before vector search.
  Why it matters: The question and stored chunks need compatible vector representations for similarity search to compare them.
• Top-two vector retrieval: Describe how the Cypher vector similarity query retrieves the top two text chunks most similar to the embedded question from Neo4j’s vector index.
  Why it matters: Retrieving the most similar chunks supplies the relevant source material for the next answer-generation step.
• Retrieved-result inspection: State that the printed search output shows the matched chunks along with each chunk’s similarity score and index, and explain that these chunks are the input for the subsequent LLM answer-generation step.
  Why it matters: Inspecting retrieval output shows what evidence was found before it is used to produce an answer.

Key terms: embedding model, embed the question, text chunks, embeddings, vector similarity search, Cypher, vector index, top two, similar chunks, matched chunks

2.2.7 Generating an answer using an LLM

This section completes the vector-search RAG flow by giving an LLM the original question and retrieved chunk text so it can stream a grounded answer.

• LLM answer-generation context: Describe how the application prepares messages for the LLM: a system message supplies instructions, while the user message includes the original question and the text from the similar chunks retrieved by vector search. Explain that the LLM uses this retrieved information to generate the answer.
  Why it matters: Supplying retrieved chunks with the question lets the LLM base its response on information found by the retriever.
• Streamed LLM response: State that the application streams the LLM result as it is generated, allowing the answer to be seen while generation is in progress.
  Why it matters: Streaming makes the generated response available incrementally instead of only after the entire answer is complete.

Key terms: system message, user message, text property, similar chunks, retriever, stream, LLM result

2.3 Adding full-text search to the RAG application to enable hybrid search

This section introduces hybrid search as a way to improve a vector-search RAG retriever by adding full-text search for production-quality results.

• Hybrid search motivation: Explain why pure vector similarity search may be insufficient for production RAG applications and why adding full-text search is proposed as an improvement to the retriever.
  Why it matters: Hybrid retrieval is introduced to seek better retrieval quality, accuracy, and performance than vector search alone can always provide.
• Retriever enhancement direction: Describe the specific enhancement introduced in this section: adding full-text search to the existing RAG application so that it can perform hybrid search.
  Why it matters: This establishes the concrete retrieval change explored in the following subsections.

Key terms: vector similarity search, full-text search, hybrid search, retriever, RAG application

2.3.1 Full-text search index

This section distinguishes keyword-based full-text search from vector similarity search and adds a Neo4j full-text index to support hybrid search.

• Full-text search matching: Explain that full-text search finds data through keyword matches rather than similarity in a vector space, and that a search term must exactly match a word in the stored data.
  Why it matters: This clarifies the keyword-based retrieval capability that complements vector similarity search in hybrid search.
• Neo4j chunk full-text index: Describe the full-text index added for hybrid search: a Neo4j index named PdfChunkFulltext on the text property of :Chunk nodes.
  Why it matters: The index makes the application’s chunk text available for the full-text part of hybrid retrieval.

Key terms: full-text search, keywords, vector space, exact match, full-text search index, Neo4j, PdfChunkFulltext, text property, :Chunk nodes, hybrid search

2.3.2 Performing hybrid search

This section explains how hybrid search combines normalized vector-similarity and full-text-search results in a Cypher query to return deduplicated top-k matches.

• Hybrid search result combination: Describe how hybrid search runs both vector similarity search and full-text search, combines their results, removes duplicates, and returns the top k results.
  Why it matters: Combining both retrieval methods can surface relevant documents that either search method alone might miss.
• Score normalization across searches: Explain why hybrid search normalizes vector and full-text scores by dividing each score by the highest score from its own search before comparing or combining results.
  Why it matters: Normalization makes scores from two different matching methods comparable when their results are combined.
• Complementary retrieval ranking: Explain what the example results show: the normalized top result remains the vector search's top result, while a different second result can appear because full-text search found a better match there.
  Why it matters: This illustrates the practical benefit of hybrid search: full-text retrieval can improve rankings for matches vector search does not rank as highly.

Key terms: hybrid search, vector similarity search, full-text search, deduplicate, top k results, union Cypher query, normalize the scores, highest score, top result, second result

2.4 Concluding thoughts

This section concludes that hybrid retrieval combines vector similarity and full-text search for better results, but remains limited by retrieving from unstructured text.

• Hybrid search trade-offs: Explain why combining vector similarity search with full-text search can produce better retrieval results than using either search method alone, while still having important limitations.
  Why it matters: Understanding both the benefit and the limits of hybrid search helps you recognize when a retriever needs more advanced strategies.
• Unstructured retrieval limitations: Describe why retrieving from unstructured data can limit hybrid-search quality: textual references may not be captured, and the retrieved surrounding context may not be enough for an LLM to understand meaning and generate good answers.
  Why it matters: These limitations motivate improving the retriever rather than assuming better search alone provides sufficient context for answer generation.

Key terms: vector similarity search, full-text search, hybrid search, retriever, unstructured data, references, surrounding context, LLMs

2.5 Summary

This summary recaps a RAG pipeline, vector and hybrid retrieval, and the limits of these retrieval approaches as data becomes more complex.