liveBook · Manning

Retrieval-augmented generation

As explained in Designing AI Systems

A retrieval-augmented generation pipeline retrieves relevant context and supplies it to a generative model when producing a response.

Retrieval-augmented generation (RAG) supplies a generative model with retrieved context before it produces a response. For an organizational assistant, this grounds answers in documents such as policies, runbooks, and product specifications rather than relying on plausible-sounding guesses. section 10.5

A basic pipeline searches the knowledge base on every request and includes the top results in the model’s context. Retrieval quality depends on the ingestion pipeline: documents are parsed, split into semantically coherent chunks, embedded, and stored with their text and metadata. At query time, the question is embedded with the same model used for ingestion and compared with stored vectors. section 5.1 section 10.5

Figure 5.1 The retrieval pipeline from document to search result. A source document enters the ingestion pipeline, where it is parsed into clean text and then broken into semantically coherent chunks. Each chunk is passed through an embedding model to produce a dense vector representation. The vectors, along with the original chunk text and metadata, are stored in a vector database. At query time, the user's question follows the same embedding process to produce a query vector, which is compared against stored vectors using similarity search. The top matching chunks are returned with their source attribution, relevance scores, and metadata, ready to be injected into the model's context as grounding information.

Follow a document from parsing and chunking through vector search and source-attributed results. section 5.1

Finding a return policy

A customer asks about returning an opened laptop. Semantic search can connect the question to a policy passage about electronics and a restocking fee, even when the wording differs. A coherent chunk keeps the relevant policy together for retrieval. section 5.1

In a service workflow, search runs before the model call; retrieved documents are added alongside conversation history and user memory. A score threshold controls what enters the context: too high can exclude relevant passages, while too low can introduce noise, waste tokens, and encourage irrelevant citations. Document identifiers preserve source attribution. section 10.5

Listing 10.6 Adding knowledge retrieval with the Data Service
def handle_message(
    message: str,
    user_id: str,
    session_id: str = None
) -> dict:
    session = platform.sessions.get_or_create(
        user_id=user_id, session_id=session_id
    )
 
    user_memory = platform.sessions.get_memory(user_id=user_id)
    memory_context = "\n".join(
        [f"- {key}: {value}" for key, value in user_memory.items()]
    )
 
    relevant_docs = platform.data.search(    #A
        index_name="company-knowledge",    #A
        query=message,    #A
        top_k=5,    
        score_threshold=0.7    #B
    )
 
    doc_context = "\n\n".join(    #C
        [f"[Source: {doc.document_id}]\n{doc.text}"    #C
         for doc in relevant_docs]    #C
    )    #C
 
    history, _ = platform.sessions.get_messages(
        session_id=session.session_id,
        limit=50
    )
 
    system_prompt = platform.models.get_prompt("claw-assistant")
    response = platform.models.chat(
        model="gpt-4o",
        messages=[
            {"role": "system",
             "content": system_prompt["content"]},
            {"role": "system",
             "content": f"User memory:\n{memory_context}\n\n"
                        f"Relevant documents:\n{doc_context}"},
            *[{"role": m.role, "content": m.content}
              for m in history],
            {"role": "user", "content": message}
        ]
    )
 
    # ... store exchange and return unchanged from listing 10.5

Inspect a workflow that searches for up to five documents, filters by score, and formats each result with its source identifier. section 10.5

Agentic RAG gives the model control over retrieval instead of searching automatically on every request. A registered search tool can accept a query formulated from the conversation, making it useful for follow-up questions whose meaning depends on earlier turns. Tool results include text, source, and score. section 10.5

Listing 10.7 Agentic RAG: the model controls retrieval
def search_knowledge(
    query: str,
    index: str
) -> list:
    results = platform.data.search(
        index_name=index,
        query=query,
        top_k=5,
        score_threshold=0.6
    )
    return [
        {"text": r.text, "source": r.document_id, "score": r.score}    #A
        for r in results    #A
    ]    #A
 
platform.tools.register(
    name="claw.knowledge.search",    #B
    description=(
        "Search the company knowledge base for relevant "
        "information. Use this when the user asks about "
        "company policies, processes, past decisions, or "
        "any organizational information. Formulate a "
        "specific search query based on the conversation "
        "context."
    ),
    parameters={
        "query": {
            "type": "string",
            "description": (
                "A specific search query. Rewrite the "
                "user's question into a search-optimized "
                "query that includes relevant context "
                "from the conversation."
            )
        },
        "index": {
            "type": "string",
            "enum": [
                "company-knowledge",
                "engineering-docs",
                "hr-policies",
                "product-specs"
            ],
            "description": "Which knowledge index to search"
        }
    },
    handler=search_knowledge    
)

Inspect the search function and tool registration used to expose knowledge retrieval to the model. section 10.5

How Designing AI Systems covers it

How deep it goes
Explains ingestion and semantic retrieval, then illustrates workflow integration, score filtering, source attribution, and a model-controlled search tool.
Examples
Uses an electronics return policy and an organizational assistant answering policy and planning questions.
What you can do after
Apply it via the Data Service to retrieve source-attributed context and tune a relevance threshold.
How it connects
Connects semantic search, chunking, embeddings, vector databases, source attribution, and tool calling.

Also explained in

Compare how 46 books explain retrieval-augmented generation →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Designing AI Systems. The section links point to the parts of the book it draws on.