chapter five

5 The Data Service: teaching AI what your organization knows

 

This chapter covers

  • Designing the Data Service to give teams searchable knowledge indexes
  • Building an ingestion pipeline that detects file formats, extracts text, and chunks documents
  • Abstracting vector storage and search with a complete pgvector implementation
  • Extending retrieval with hybrid vector and keyword search
  • Exposing the Data Service through the gRPC contract and platform SDK

An AI assistant that remembers your conversation but doesn't know your company's policies, products, or procedures is still going to make things up. It will hallucinate confidently about return windows, invent product features, and cite policies that don't exist. Conversational memory, which we built in chapter 4, is only half the story. The other half is grounding: connecting AI applications to organizational knowledge so that responses reflect reality rather than plausible guesses.

The Data Service provides this grounding. It gives teams a way to turn documents such as company policies, product documentation, support articles, technical manuals, and internal wikis into searchable knowledge without each team building its own parsing, chunking, embedding, and storage pipeline. The platform lets teams create isolated knowledge indexes, choose how their documents get chunked and embedded, and search across them without worrying about the infrastructure underneath.

5.1 From documents to searchable knowledge

5.2 Indexes: organizing knowledge

5.2.1 Why isolation matters

5.2.2 Index configuration

5.2.3 Index operations

5.3 Ingestion pipeline: from raw files to vectors

5.3.1 The challenge of diverse formats

5.3.2 Pipeline architecture

5.3.3 Format detection and text extraction

5.3.4 Metadata: the filtering foundation

5.3.5 Chunking: breaking text into retrievable pieces

5.3.6 Generating embeddings

5.3.7 Document lifecycle

5.3.8 Document management

5.3.9 Asynchronous ingestion

5.4 Vector storage and search

5.4.1 Vector store interface

5.4.2 Choosing a vector store backend

5.4.3 The pgvector implementation

5.4.4 Search orchestration

5.5 Hybrid search: combining vectors with keywords

5.5.1 Adding keyword search to the platform

5.5.2 PostgreSQL keyword search implementation

5.5.3 Merging results: Reciprocal Rank Fusion

5.5.4 Putting it together

5.6 Service contract and complete retrieval flow

5.7 Summary