Information retrieval

As explained in Data Storytelling with Altair and AI

Information retrieval is the process of searching for and returning relevant information from a collection or external knowledge source.

Information retrieval searches an external knowledge source, such as a text corpus or structured database, and returns context that can improve generated answers. In retrieval-augmented generation (RAG), retrieval precedes text generation and supplies relevant facts. section 7.4.2

A practical retrieval pipeline moves through these stages:

• Load a product webpage and split its text into overlapping chunks. section 7.4.2
• Represent chunks as embeddings and persist them in a Chroma vector database. section 7.4.2
• Query the store through LangChain, then pass retrieved context to an LLM. section 7.4.2

Product commentary

A RAG application queries a smartphone description stored from an HTML page and generates a concise answer about the New SmartX 2023, returning source documents alongside the result. section 7.4.2

Figure 7.13 A RAG-based system

Use the architecture diagram to trace the flow from external product text through retrieval to generated commentary. section 7.4.2

How Data Storytelling with Altair and AI covers it

How deep it goes

Explains retrieval through loading, chunking, embedding, storage, prompting, querying, and returned sources, without developing retrieval algorithms internally.

Examples

Demonstrates retrieval over an HTML smartphone description using LangChain, Chroma, OpenAI embeddings, and a RetrievalQA chain.

What it assumes

Assumes familiarity with LLM applications and introduces the preparation needed to load, split, vectorize, store, and query text.

What you can do after

Build it from scratch with LangChain and Chroma to answer product questions using retrieved webpage context.

How it connects

Connects information retrieval with retrieval-augmented generation, embeddings, vector databases, prompt templates, and large language models.

liveBookmanning.com

Read Data Storytelling with Altair and AI on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Introduction to Generative AI, Second EditionRetrieval is the process of finding relevant information from an external collection in response to a query or representation.
• 

  liveBookmanning.com
  Effective Conversational AIThe retrieval phase is the stage of a retrieval-augmented generation process that finds and obtains information relevant to an input.
• 

  liveBookmanning.com
  Grokking AI ApplicationsThe process of finding and obtaining relevant information from a data source for use in an AI application.
• 

  liveBookmanning.com
  MongoDB 8.0 in Action, Third EditionInformation retrieval is the process of finding relevant information from a collection of stored data in response to an information need.
• 

  liveBookmanning.com
  Natural Language Processing in Action, Second EditionFull-text search retrieves documents by searching their complete textual contents rather than only structured fields.
• 

  liveBookmanning.com
  Microsoft Azure in ActionRetrieval is the process of selecting relevant stored information to provide as context for a language model.
• 

  liveBookmanning.com
  Elasticsearch in Action, Second EditionSearching textual content using tokenization, matching, relevance scoring, or related linguistic analysis.
• 

  liveBookmanning.com
  Build a Website with ChatGPTSearch is the process of finding records whose data matches a user-supplied word or phrase.
• 

  liveBookmanning.com
  Ensemble Methods for Machine LearningInformation retrieval is the field concerned with finding and ranking relevant information items in response to queries.
• 

  liveBookmanning.com
  Building Reliable AI SystemsRetrieval is the process of finding relevant information from an external collection to supply as model context.

Compare how 11 books explain information retrieval →

Related concepts

• retrieval-augmented generationused by · Retrieval-augmented generation is a technique that retrieves relevant information from external sources and supplies it as context to a generative model when producing an output.
• hybrid searchused by · Hybrid search is an information-retrieval technique that combines multiple retrieval methods or relevance signals to find or rank relevant results.
• similarity searchincludes · Similarity search is a retrieval technique that finds or ranks items by comparing their representations with a query according to a similarity or distance measure.
• chunkingused by · Chunking is the process of dividing content or data into smaller units for processing, retrieval, or organization.
• agentic RAGused by · Agentic RAG is a retrieval-augmented generation approach in which an agent dynamically decides how and when to retrieve information, often iterating or using tools before generating a response.

This explainer was generated by Manning's knowledge graph from the text of Data Storytelling with Altair and AI. The section links point to the parts of the book it draws on.