liveBook · Manning

Chapter 5: The Data Service: teaching AI what your organization knows

What you will learn in chapter 5 of Designing AI Systems

This chapter introduces the Data Service as platform infrastructure that grounds AI responses in searchable organizational knowledge rather than relying only on conversational memory.

After this chapter you will be able to

What it builds on

Section by section

Introduction

This chapter introduces the Data Service as platform infrastructure that grounds AI responses in searchable organizational knowledge rather than relying only on conversational memory.

Key terms: grounding, Data Service, organizational knowledge, conversational memory, hallucinate, unstructured, semantic matching, natural language, conversation history, metadata filtering

5.1 From documents to searchable knowledge

This section explains how the Data Service turns organizational documents into searchable knowledge for retrieval-augmented AI applications.

Key terms: semantic search, keyword matching, similarity matching, ingestion pipeline, chunks, chunking, semantic coherence, embeddings, dense vectors, vector space

5.2 Indexes: organizing knowledge

This section introduces an index as the Data Service unit for organizing related knowledge into isolated, independently configured collections.

Key terms: index, Data Service, documents, search, isolated collection, independent body of knowledge, named, embedding configuration, search behavior

5.2.1 Why isolation matters

This subsection explains why separate indexes isolate teams’ knowledge and allow embedding and chunking choices to fit each collection safely.

Key terms: index, index isolation, Data Service, embedding model, embedding configuration, index-level setting, allowed list, index creation, chunking strategy, FAQ entries

5.2.2 Index configuration

This section defines an index’s configuration and explains why embedding settings are immutable, how to migrate models safely, and how chunking and metadata schemas protect retrieval quality.

Key terms: name, embedding_model, embedding_dimensions, index, re-ingest, validation queries, rollback, index abstraction, embedding configuration, chunking_strategy

5.2.3 Index operations

This section defines index lifecycle operations, the metadata returned for indexes, and why creation, ingestion, naming, and ownership are treated as separate operational concerns.

Key terms: create, list, delete, index management operations, Index, document count, total chunks, created_at, last_ingested_at, index creation

5.3 Ingestion pipeline: from raw files to vectors

This section introduces the ingestion pipeline that turns source documents into chunk embeddings stored for semantic search within an index.

Key terms: ingestion pipeline, source file, text extraction, chunks, embeddings, vector store

5.3.1 The challenge of diverse formats

This subsection explains why a platform ingestion pipeline must reliably extract text from many document formats while remaining reusable and extensible for unusual sources.

Key terms: ingestion pipeline, clean, structured text, PDFs, Markdown, HTML pages, Word documents, shared ingestion pipeline, platform-level ingestion pipeline, RAG application, parsers

5.3.2 Pipeline architecture

This section traces how a staged ingestion pipeline converts a raw document into embedded, metadata-bearing chunks that can be searched.

Key terms: ingestion pipeline, format detection, parser, text extraction, metadata, chunks, clean text, structural information, headings, paragraphs

5.3.3 Format detection and text extraction

This section explains how the ingestion pipeline detects document formats, extracts structured text through format-specific parsers, and remains extensible through custom parser registration.

Key terms: DocumentParser, ExtractedDocument, sections, structure-aware chunking, fixed-size chunking, format detection, file headers, file extensions, magic byte signatures, parser routing

5.3.4 Metadata: the filtering foundation

This section explains how document metadata is captured, standardized, and used to filter, attribute, rank, and audit retrieved organizational knowledge.

Key terms: metadata, automatic metadata, caller-supplied metadata, ingestion timestamp, custom_metadata, metadata filtering, vector search, audience, customer, internal engineering runbook

5.3.5 Chunking: breaking text into retrievable pieces

This section explains how the Data Service splits extracted documents into retrievable chunks, selects chunking strategies, and preserves context for accurate vector retrieval.

Key terms: chunking, chunk size, chunk overlap, retrieval quality, Chunk, text, heading, start_offset, end_offset, metadata

5.3.6 Generating embeddings

This section explains how the Data Service turns text chunks into consistently comparable embeddings through the Model Service, balancing retrieval quality, performance, and operational concerns.

Key terms: embedding model, index configuration, chunks, queries, vectors, mathematical spaces, dimensionality, floating-point values, semantic relationships, storage costs

5.3.7 Document lifecycle

This section explains how the data service safely updates documents through replace-on-reingest, atomic ingestion, and index-level swaps for consistent bulk updates.

Key terms: replace-on-reingest, diff-based updates, chunks, vectors, re-embedding, ingestion method, document_id, format detection, text extraction, chunking

5.3.8 Document management

This section defines Data Service operations for listing, inspecting, and safely deleting indexed documents without exposing the vector store directly.

Key terms: list_documents, get_document, delete_document, source URI, ingestion timestamp, chunk count, cascading delete, atomic, document record, chunks

5.3.9 Asynchronous ingestion

This section introduces asynchronous ingestion so large or batch document uploads return immediately and are processed in the background with trackable job status.

Key terms: synchronous ingestion, asynchronous ingestion, background, ingestion pipeline, job_id, queued, processing, completed, failed, progress

5.4 Vector storage and search

This section introduces vector storage as a backend-independent abstraction for storing embedding vectors and searching them by similarity.

Key terms: vector store, storage abstraction, vector database backend, consistent interface, embeddings, similarity search, session storage, vector storage, sequential messages, ID

5.4.1 Vector store interface

This section defines a backend-agnostic vector store interface for storing embedded chunks, deleting scoped content, and retrieving ranked, filtered search results.

Key terms: VectorStore, insert, chunks, embeddings, index_name, document_id, delete_by_document, delete_index, owner, search

5.4.2 Choosing a vector store backend

This section compares vector-store backend options and explains why the platform implements pgvector behind a replaceable VectorStore abstraction.

Key terms: vector store, database extensions, purpose-built vector databases, similarity search, high-dimensional similarity search, pgvector, PostgreSQL, vector column types, similarity operators, indexing

5.4.3 The pgvector implementation

This section implements the VectorStore interface with PostgreSQL and pgvector, covering chunk storage, vector indexing, and filtered similarity search.

Key terms: pgvector, vector column, embedding column, chunk_text, JSONB, document_id, IVFFlat, Inverted File Flat, HNSW, Hierarchical Navigable Small World

5.4.4 Search orchestration

This section shows how the Data Service turns a plain-text query into vector-store search results by coordinating index configuration, embedding generation, and vector search.

Key terms: Data Service, search method, index configuration, embedding generator, vector store, filters, embedding model, query, chunk vectors, vector similarity

5.5 Hybrid search: combining vectors with keywords

This section explains why hybrid search combines semantic vector search with keyword search to cover both meaning-based and exact-match queries.

Key terms: semantic search, embeddings, dense vectors, vector similarity search, keyword search, traditional text search, exact matches, partial matches, boolean queries, phrase matching

5.5.1 Adding keyword search to the platform

This section extends the VectorStore interface with an optional keyword retrieval method so the platform can support hybrid search without disrupting vector-only backends.

Key terms: VectorStore, keyword_search, search, raw query string, embedding vector, index scoping, NotImplementedError, abstract method, backend, FAISS

5.5.2 PostgreSQL keyword search implementation

This section implements PostgreSQL keyword search over document chunks using generated tsvectors, while explaining how its ranking differs from BM25 and why it is sufficient for hybrid retrieval.

Key terms: BM25, Best Matching 25, term frequency, inverse document frequency, document length, Elasticsearch, tsvector, chunk_text, GENERATED ALWAYS AS, stop words

5.5.3 Merging results: Reciprocal Rank Fusion

This section explains how Reciprocal Rank Fusion combines vector and keyword search rankings without comparing their incompatible raw scores, and why it is usually the practical hybrid-search default.

Key terms: cosine similarity, keyword search, score distributions, normalization, heterogeneous search results, Reciprocal Rank Fusion, RRF, rank, 1/(k + rank), combined score

5.5.4 Putting it together

This section combines vector and keyword retrieval into one hybrid-search operation that fuses parallel ranked results into a consistent response.

Key terms: hybrid_search, vector search, keyword search, Model Service, cosine similarity, tsquery, oversampling, candidates, RRF, recall

5.6 Service contract and complete retrieval flow

This section exposes the Data Service through a gRPC contract and thin SDK client, clarifying how retrieval capabilities compose into RAG while workflows retain application-level decisions.

Key terms: Data Service, gRPC contract, index management, document ingestion, search, hybrid search, DataClient, gRPC, RPC, stub

5.7 Summary

This summary explains how the Data Service turns organizational documents into independently managed, searchable knowledge that workflows retrieve to ground AI responses.

Where this is used later

Concepts explained in this chapter

← Chapter 4: The Session Service: Teaching your AI to remember Chapter 6: Tools and Guardrails: Enabling safe, managed AI actions →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.