Information retrieval
Information retrieval is the process of finding and selecting information relevant to a query or information need from a collection.
Introduction to Generative AI, Second Edition
Retrieval is the process of finding relevant information from an external collection in response to a query or representation.
- How it is used
- Uses retrieval to ground language-model responses in external information beyond static training data.
- How deep it goes
- Explains query, retrieval, summarization, and vector similarity at an overview level without implementation details.
- Examples
- Illustrates retrieval with a factual search query and vegan restaurant recommendations using previous preferences and Yelp reviews.
- What you can do after
- Evaluate and understand how retrieved sources can support an answer while recognizing that queries, summaries, and sources require checking.
Read how Introduction to Generative AI, Second Edition explains information retrieval →
Vibe Engineering
Retrieval is the process of selecting relevant information from an indexed or accessible knowledge source.
- How it is used
- Supplies coding agents with relevant repository, documentation, or tool context without loading everything at once.
- How deep it goes
- Explains embedding-based indexing and retrieval, contrasts live code search, and gives manual scoping and source-quality checks.
- Examples
- Uses an authentication handler and tests, file-picker documentation, and deferred tool discovery as retrieval scenarios.
- What you can do after
- Operate/configure retrieval by scoping context manually, curating sources, and checking critical tool discovery.
Effective Conversational AI
The retrieval phase is the stage of a retrieval-augmented generation process that finds and obtains information relevant to an input.
- How it is used
- Finds document passages for nuanced, long-tail chatbot questions that predefined intents may not answer well.
- How deep it goes
- Explains retrieval through document preparation, chunking, embeddings, and vector storage, while distinguishing retrieved passages from generated answers.
- Examples
- Contrasts a generic side-effects answer with passages addressing a vaccine, ibuprofen, pain, and blood-pressure medication.
- What you can do after
- Evaluate/understand when retrieved passages suffice and when RAG better reduces the work of assembling an answer.
Read how Effective Conversational AI explains information retrieval →
Retrieval Augmented Generation, The Foundational Ideas
Retrieval finds candidate documents or passages from a knowledge base in response to a query.
- How it is used
- Fetches candidate evidence for ranking and grounded answer generation, including information unavailable in a model’s training data.
- How deep it goes
- Explains offline indexing and online candidate retrieval, then surveys search techniques without specifying a complete implementation here.
- Examples
- Uses a warranty question, changing organizational knowledge, and a historical retrieval timeline to distinguish search needs and approaches.
- What you can do after
- Evaluate/understand whether missing content calls for indexing or retrieval changes and whether buried candidates call for ranking changes.
Read how Retrieval Augmented Generation, The Foundational Ideas explains information retrieval →
Grokking AI Applications
The process of finding and obtaining relevant information from a data source for use in an AI application.
- How it is used
- Supplies relevant document chunks to answer generation when a RAG user asks a question requiring stored knowledge.
- How deep it goes
- Describes selecting a few candidate chunks from a vector database and illustrates distance-based matching.
- Examples
- Uses a question matched against six pet-related chunks, with Euclidean distances showing which chunks are closest.
- What you can do after
- Understand how retrieved document chunks provide extra knowledge for an LLM’s generated answer.
Read how Grokking AI Applications explains information retrieval →
AI Recommender Systems
A candidate-generation technique that finds potentially relevant items or documents from a larger collection.
- How it is used
- Generates a broad candidate pool before filtering, scoring, and final ordering in recommender pipelines.
- How deep it goes
- Explains pipeline placement, interchangeable implementations, and recall-based evaluation; defers construction of the pool-size measurement.
- Examples
- Uses a popular-content carousel, co-purchased products, and a four-stage recommender function to illustrate candidate generation.
- What you can do after
- Build it from scratch as a pipeline stage, swap retrieval methods, and diagnose missed candidates with recall at multiple pool sizes.
Read how AI Recommender Systems explains information retrieval →
MongoDB 8.0 in Action, Third Edition
Information retrieval is the process of finding relevant information from a collection of stored data in response to an information need.
- How it is used
- Finds and ranks relevant documents for application search and retrieves external information to ground RAG responses.
- How deep it goes
- Explains search modes, inverted indexing, operational trade-offs, and a worked local query; stops short of implementing a search engine.
- Examples
- Demonstrates fuzzy inspection-record search with sector and city filters, relevance scores, and a three-result limit.
- What you can do after
- Apply it via Atlas Search by creating a local index and querying, filtering, and ranking collection results.
Read how MongoDB 8.0 in Action, Third Edition explains information retrieval →
Natural Language Processing in Action, Second Edition
Full-text search retrieves documents by searching their complete textual contents rather than only structured fields.
- How it is used
- Retrieves relevant text quickly and supplies grounded documents for search, question answering, and language-model workflows.
- How deep it goes
- Explains forward and inverted indexes, lookup scaling, TF-IDF, trigrams, and integration with semantic search, while deferring full implementation details.
- Examples
- Uses textbook indexes, cat-versus-cats matching, typo-tolerant trigrams, and large-corpus search as concrete examples.
- What you can do after
- Build a retrieval component with TF-IDF vectors, inverted indexing, character trigrams, and semantic-search or LLM integration.
Read how Natural Language Processing in Action, Second Edition explains information retrieval →
Human-in-the-Loop Machine Learning: Active learning and annotation for human-centered AI
The task of finding and retrieving relevant information from a collection in response to an information need.
- How it is used
- Supports search and recommendation while guiding human annotation of ranked results.
- How deep it goes
- Explains ranked relevance, DCG weighting, annotator scoring, and model-assisted sampling without detailing retrieval algorithm implementation.
- Examples
- Uses online product search and basketball queries to illustrate retrieval workflows and sparse relevant results.
- What you can do after
- Evaluate ranked retrieval results and annotator rankings using relevance and discounted cumulative gain.
Microsoft Azure in Action
Retrieval is the process of selecting relevant stored information to provide as context for a language model.
- How it is used
- Supplies relevant pieces of oversized books or conversations to a language model with a limited text window.
- How deep it goes
- Defines retrieval through an analogy but gives no search procedure or configuration steps.
- Examples
- Uses a fixed-size suitcase and an oversized book or conversation to illustrate selective fetching.
- What you can do after
- Evaluate/understand it as an option for giving a language model relevant context when the full input will not fit.
Read how Microsoft Azure in Action explains information retrieval →
Elasticsearch in Action, Second Edition
Searching textual content using tokenization, matching, relevance scoring, or related linguistic analysis.
- How it is used
- Supports application search across varied content providers, websites, mobile applications, and server-side services.
- How deep it goes
- Explains analysis, tokenization, normalization, and inverted-index matching, while deferring detailed query construction.
- Examples
- Uses enterprise content providers, application search, French analyzers, root words, synonyms, and spelling mistakes as examples.
- What you can do after
- Operate retrieval through Kibana or REST clients to execute queries and return matching data from Elasticsearch.
Read how Elasticsearch in Action, Second Edition explains information retrieval →
Data Storytelling with Altair and AI
Information retrieval is the process of searching for and returning relevant information from a collection or external knowledge source.
- How it is used
- Uses external product information to ground generated commentaries and annotations in data such as websites, reviews, or real-time information.
- How deep it goes
- Explains retrieval through loading, chunking, embedding, storage, prompting, querying, and returned sources, without developing retrieval algorithms internally.
- Examples
- Demonstrates retrieval over an HTML smartphone description using LangChain, Chroma, OpenAI embeddings, and a RetrievalQA chain.
- What you can do after
- Build it from scratch with LangChain and Chroma to answer product questions using retrieved webpage context.
Read how Data Storytelling with Altair and AI explains information retrieval →
Build a Website with ChatGPT
Search is the process of finding records whose data matches a user-supplied word or phrase.
- How it is used
- Supports interactive web pages by replacing displayed records with data matching user-entered text.
- How deep it goes
- Explains the text-box workflow and field selection, but defers the generated JavaScript implementation details.
- Examples
- Uses a course catalog, a Search text box, an article element, and JSON records as concrete artifacts.
- What you can do after
- Build it from scratch by prompting for HTML and JavaScript that search JSON records while typing.
Read how Build a Website with ChatGPT explains information retrieval →
Ensemble Methods for Machine Learning
Information retrieval is the field concerned with finding and ranking relevant information items in response to queries.
- How it is used
- Frames search as matching queries to documents and ranking results by relevance.
- How deep it goes
- Explains retrieval through matching, ranking, and three-class relevance modeling, then stops at model comparisons.
- Examples
- Uses web search, legal archives, journal databases, related articles, and query-document relevance classes as examples.
- What you can do after
- Apply information retrieval via LightGBM or XGBoost to classify query-document relevance and compare model performance.
Read how Ensemble Methods for Machine Learning explains information retrieval →
LLM Customization and Fine-Tuning
The process of finding the most relevant text chunks for a query based on embedding similarity.
- How it is used
- Finds text chunks relevant to a query by comparing their embeddings.
- How deep it goes
- Explains vector-space search, similarity metrics, top-k sorting, and embedding-size tradeoffs.
- Examples
- Uses revenue and earnings paragraphs as nearby matches, contrasted with an unrelated lunch paragraph.
- What you can do after
- Understand retrieval as finding the vector-space neighbors of an embedded query.
Read how LLM Customization and Fine-Tuning explains information retrieval →
Machines that Think
Retrieval is the computational process of selecting relevant information from an external collection in response to a query.
- How it is used
- Selects relevant passages from external knowledge bases for insertion into a model’s prompt during generation.
- How deep it goes
- Explains query embeddings, similarity ranking, and prompt insertion, then distinguishes retrieval from guaranteed grounding.
- Examples
- Uses legal archives, medical databases, corporate repositories, and query-to-document matching as examples.
- What you can do after
- Evaluate retrieval as a way to expose relevant evidence while recognizing that it cannot guarantee grounded generation.
Read how Machines that Think explains information retrieval →
Building Reliable AI Systems
Retrieval is the process of finding relevant information from an external collection to supply as model context.
- How it is used
- Supplies relevant external information as context for retrieval-augmented generation.
- How deep it goes
- Traces document and query embedding through vector storage and similarity search, without detailing search algorithms.
- Examples
- Uses a health-insurance chatbot that misses policy coverage information despite having access to the documents.
- What you can do after
- Diagnose how embedding quality can cause a retrieval pipeline to miss relevant context.
Read how Building Reliable AI Systems explains information retrieval →
Also covered in
- Enterprise RAGIn early access (MEAP)
- AI Model EvaluationIn early access (MEAP)
- Essential GraphRAG
- PostgreSQL Mistakes and How to Avoid Them
- Learn Haskell by Example
- Context EngineeringIn early access (MEAP)
- Building LLM Applications with DSPy
- Inside AI
- Knowledge Graphs and LLMs in Action
- Just Use Postgres!
- How Computers Make Books
- Generative AI in Action
What to know first
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- retrieval-augmented generationRetrieval-augmented generation is a technique that retrieves relevant information from external sources and supplies it as context to a generative model when producing an output.
- promptA prompt is text or other input supplied to an AI model to provide instructions, context, or other information that guides its response or generated output.
- context windowA context window is a bounded span of sequence elements or contextual information that a model or processing operation can consider together.
Related concepts
- retrieval-augmented generationused by · Retrieval-augmented generation is a technique that retrieves relevant information from external sources and supplies it as context to a generative model when producing an output.
- hybrid searchused by · Hybrid search is an information-retrieval technique that combines multiple retrieval methods or relevance signals to find or rank relevant results.
- chunkingused by · Chunking is the process of dividing content or data into smaller units for processing, retrieval, or organization.
- agentic RAGused by · Agentic RAG is a retrieval-augmented generation approach in which an agent dynamically decides how and when to retrieve information, often iterating or using tools before generating a response.
- case-based reasoningpart of · Case-based reasoning is an artificial-intelligence technique that solves new problems by retrieving, adapting, and revising solutions from similar past cases.
- context injectionalternative to · Context injection is the technique of adding externally supplied information to a model’s input so it can inform the model’s generated response.
Questions about information retrieval
What is information retrieval?
Information retrieval is the process of finding and selecting information relevant to a query or information need from a collection.
Which Manning books explain information retrieval?
Introduction to Generative AI, Second Edition, Vibe Engineering, Effective Conversational AI, Retrieval Augmented Generation, The Foundational Ideas, Grokking AI Applications, AI Recommender Systems, MongoDB 8.0 in Action, Third Edition, Natural Language Processing in Action, Second Edition and 9 more.
What should I know before learning information retrieval?
Manning books that teach information retrieval usually introduce language model, retrieval-augmented generation, prompt and context window first.
What is the difference between information retrieval and context injection?
Information retrieval is the process of finding and selecting information relevant to a query or information need from a collection. Context injection is the technique of adding externally supplied information to a model’s input so it can inform the model’s generated response.
What is information retrieval part of?
Information retrieval is part of case-based reasoning. Case-based reasoning is an artificial-intelligence technique that solves new problems by retrieving, adapting, and revising solutions from similar past cases.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
















