liveBook · Manning

Generation pipeline

As explained in Context Engineering

A generation pipeline is an online workflow that retrieves relevant information at query time and uses it to produce a model response.

A generation pipeline is the online, synchronous path in retrieval-augmented generation (RAG): it searches external knowledge at query time, adds relevant context to a prompt, and generates a response. Unlike the offline indexing pipeline, its operations run while the user waits and must fit an interactive latency budget. section 3.2

A typical request passes through these stages:

  • Optionally transform the query by rewriting, expanding, or decomposing it to improve alignment with the indexed corpus. section 3.2.2
  • Retrieve matching chunks from the index. In dense retrieval, the query is embedded with the same model used for indexing and compared with stored vectors. section 3.2.2
  • Select the top-K candidates and, where appropriate, rerank them with a more precise model. Top-K balances useful coverage against irrelevant context. section 3.2.2
  • Insert the retrieved context into a prompt and send it to the model. A prompt can instruct the model to answer only from the supplied context. section 3.10.1

Minimal sparse retrieval

A Python RAG implementation indexes documents with a TF-IDF vectorizer. At query time, it transforms the question with that same vectorizer, ranks document vectors by cosine similarity, and places the highest-scoring matches in the model prompt. This makes the mechanics inspectable without a dense vector database. section 3.10.1

Listing 3.1 Minimal RAG in Python: TF-IDF indexing, similarity-based retrieval, and augmented generation
corpus = [doc["text"] for doc in documents] #1
vectorizer = TfidfVectorizer(stop_words="english")
doc_vectors = vectorizer.fit_transform(corpus) #2

def retrieve(query: str, k: int = 3) -> List[Dict[str, Any]]: #3
    query_vec = vectorizer.transform([query])
    similarities = cosine_similarity(query_vec, doc_vectors)[0]
    top_indices = np.argsort(similarities)[::-1][:k]
    results: List[Dict[str, Any]] = []
    for idx in top_indices:
        results.append({
            "id": documents[idx]["id"],
            "title": documents[idx]["title"],
            "score": float(similarities[idx]),
            "text": documents[idx]["text"],
        })
    return results

def query_model_with_rag(question: str,
                k: int = 3,
                model: str = "gpt-4o-mini") -> str: #4
    retrieved_docs = retrieve(question, k=k)
    context = build_context(retrieved_docs)
    prompt = (
        "You are a helpful assistant ...\n"
        "Documentation:\n"
        f"{context}\n\n"
        f"User question: {question}"
    )
    # Query model (see full source code in GitHub)

Inspect the Python listing for vectorization, similarity scoring, top-K selection, and augmented generation. section 3.10.1

Grounding does not guarantee a reliable answer. Source freshness and authority, chunking, ranking, permission filters, and prompt assembly can each affect whether the model receives appropriate evidence. Query transformation, retrieval, reranking, and inference also contribute to request latency. section 3.2 section 3.2.2

How Context Engineering covers it

How deep it goes
Explains query transformation, retrieval, top-K, and reranking, and works through a minimal sparse-retrieval implementation.
Examples
Uses a Python implementation with TF-IDF document vectors, cosine similarity ranking, and an OpenAI model.
What you can do after
Apply it using Python, TF-IDF retrieval, and the OpenAI API to produce answers from retrieved context.
How it connects
Connects retrieval-augmented generation with indexing pipelines, embeddings, vector databases, and cosine similarity.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Context Engineering. The section links point to the parts of the book it draws on.