Chapter 5: The Data Service: teaching AI what your organization knows
This chapter introduces the Data Service as platform infrastructure that grounds AI responses in searchable organizational knowledge rather than relying only on conversational memory.
After this chapter you will be able to
- Organizational knowledge grounding. Explain why AI applications need grounding in organizational knowledge in addition to conversational memory, and how the Data Service helps responses reflect real company policies, products, and procedures rather than plausible guesses.
- Unstructured retrieval problem. Describe why retrieving organizational knowledge differs from looking up conversation records: documents are unstructured, and retrieval must find passages relevant to a natural-language question even when they use different wording.
- Data Service retrieval pipeline. Trace the high-level Data Service pipeline: it turns documents into meaningful chunks, converts chunks into mathematical representations of meaning, stores them for efficient similarity search, and combines semantic search with metadata filtering.
- Semantic search rationale. Explain why semantic search is more reliable than keyword matching for connecting a user question to relevant organizational knowledge. Describe how semantic search matches meaning even when the question and document use different words.
- Chunk embedding retrieval pipeline. Describe the path from a source document to a search result: parse document text, divide it into semantically coherent chunks, convert chunks and queries into embeddings, and retrieve similar chunk vectors from a vector store. Explain why chunk-level comparison is used instead of comparing a short query with an entire large document.
- Metadata-aware search results. Describe why retrieval returns more than raw chunk text: it includes the source document, a match-confidence score, and metadata. Explain how metadata filtering restricts semantically similar results to the appropriate domain, such as customer-facing rather than internal content.
- Index as knowledge unit. Explain that an index is the Data Service organizational unit used before document ingestion or search to group related organizational knowledge.
- Knowledge isolation boundaries. Describe how indexes keep unrelated knowledge independent, so separate domains such as product documentation, HR policies, and engineering runbooks can use separate collections.
- Index isolation. Explain how separate indexes keep each team’s searches confined to the relevant document collection, such as Support searching troubleshooting guides and Legal searching compliance documents.
- Per-index embedding configuration. Describe why embedding-model configuration belongs to an individual index rather than being global: different content domains can use models suited to their material, while the platform can allow only approved models at index creation.
- Index embedding configuration. Describe the index fields that identify an index and determine how its text becomes vectors, including name, embedding_model, and embedding_dimensions. Explain why the embedding model and dimensions are fixed when the index is created.
- Embedding model migration. Trace the safe process for changing embedding models: create a new index, re-ingest documents, validate retrieval with queries, shift traffic after validation, and retain the old index for rollback. Explain how separate indexes let different embedding models coexist without interference.
- Index lifecycle operations. Describe the three index lifecycle operations: creating, listing, and deleting indexes, and explain that they are exposed through an abstract management interface.
- Index status metadata. Explain what the Index object reports, including its configuration, document count, total chunks, creation time, and last ingestion time. Describe how last_ingested_at helps teams verify that scheduled or update-triggered ingestion is keeping an index current.
- Creation-ingestion separation. Explain that creating an index sets up an empty container with a name, embedding configuration, chunking strategy, and storage structures, but adds no documents. Distinguish this one-time setup from ongoing ingestion, which supplies documents through the ingestion pipeline on a schedule or after updates.
- Document ingestion pipeline. Describe the end-to-end ingestion pipeline: it takes a source file, extracts clean text, splits that text into chunks, generates embeddings for the chunks, and stores the results in a vector store.
- Diverse format extraction challenges. Describe why extracting clean, structured text from organizational documents is difficult across PDFs, Word documents, HTML pages, and Markdown. Explain that source formats include presentation or surrounding material that can distort, obscure, or pollute the actual document content.
- Shared platform parsing. Explain the rationale for a shared, platform-level ingestion pipeline rather than separate parsers in every RAG application team. State that the platform solves common parsing problems once, so teams avoid duplicated work and parser fixes benefit every index that uses them.
- Document ingestion pipeline. Trace the end-to-end ingestion pipeline from a source file through format detection, parsing, metadata capture, chunking, embedding, and vector-store storage. Explain that each stage has a distinct responsibility and passes its output to the next stage.
- Chunk-level metadata propagation. Explain the two sources of metadata captured during ingestion and how it travels with every derived chunk. Distinguish document-derived metadata such as filename, page count, word count, and detected language from caller-supplied metadata such as department, document type, author, version, and tags, and state that it enables filtered search.
- Structured extraction model. Describe how a DocumentParser produces an ExtractedDocument containing sections, headings, hierarchy, and text derived from those sections. Explain how this structure supports both structure-aware chunking and fixed-size chunking.
- Format detection and routing. Explain that the pipeline identifies a file's real format by inspecting file headers first and falling back to extensions when a format lacks a reliable magic-byte signature, then routes the file to the appropriate parser.
- Custom parser extensibility. Explain how a team adds a new format by implementing a DocumentParser subclass and registering it in the parser registry through the SDK, leaving chunking, embedding, storage, and search unchanged. State why tracking the parser version used for ingestion identifies documents that may need re-ingestion after parser logic changes.
- Document metadata sources. Describe document metadata as attributes beyond text content, distinguishing metadata extracted during parsing from metadata supplied by the caller at ingestion.
- Audience-based metadata filtering. Explain how metadata filtering narrows vector-search results using attributes such as audience, so a workflow can retrieve customer material without exposing an internal engineering document that discusses the same procedure.
- Chunk quality tradeoffs. Explain why chunking divides extracted text into retrievable pieces and how chunk size, context, and boundaries affect retrieval quality. Describe why chunks that are too large, too small, or split in the middle of a sentence produce poor results.
- Chunk representation and strategy interface. Describe the information a Chunk carries and how the common ChunkingStrategy interface makes chunking pluggable. Explain that index-configured size and overlap are supplied to a strategy, which converts an ExtractedDocument into a list of Chunk objects without requiring pipeline changes.
- Built-in chunking strategies. Distinguish fixed-size chunking, recursive splitting, and structure-aware chunking, including when each is appropriate. Explain why recursive splitting is the platform default and how overlap or document structure preserves meaningful context.
- Index embedding model consistency. Explain why every chunk in an index, and every query searched against it, must use the same embedding model. Describe how index-level configuration enforces this consistency while allowing teams to choose the model appropriate for their content.
- Model Service embedding delegation. Explain why the Data Service generates embeddings through the Model Service rather than calling embedding providers directly. Distinguish embedding generation, which takes text and returns vectors, from chat completion, which takes messages and returns generated text, while identifying the shared platform infrastructure concerns.
- Replace-on-reingest strategy. Explain how replace-on-reingest updates a document when it is ingested again with the same identifier, and justify why the platform favors it over diff-based updates.
- Atomic and bulk re-ingestion safety. Explain why replace-on-reingest must be atomic for one document, and describe the new-index-and-traffic-swap approach when a bulk re-ingestion cannot tolerate a mixed index state.
- Document management operations. Describe the three Data Service document management operations and what each returns or removes. Explain that list_documents shows metadata for every document in an index, get_document retrieves one document's metadata by ID, and delete_document removes a document and its indexed content.
- Atomic cascading deletion. Explain why delete_document performs an atomic cascading delete across the document record, its chunks, and corresponding vector-store vectors. Describe how partial deletion can leave orphaned vectors that silently corrupt search results.
- Asynchronous ingestion rationale. Explain why synchronous ingestion becomes impractical for very large documents or batches, and why the platform should return immediately while parsing, chunking, embedding, and storage run in the background.
- Ingestion job tracking. Describe the asynchronous ingestion job interface: submitting a file returns a job ID immediately, and teams poll the job to observe its state, progress, successful document ID, or failure error.
- Vector storage abstraction. Explain why the Data Service uses a consistent vector-store abstraction instead of tying ingestion and retrieval to one vector database backend. Describe how this lets teams select a backend for their scale and performance needs while the rest of the Data Service continues to work the same way.
- Vector and session storage contrast. Distinguish vector storage from session storage in terms of the data and lookup operations involved. State that session storage handles sequential messages retrieved by ID, whereas vector storage handles floating-point embedding arrays retrieved through similarity search, even though both use the same abstraction pattern.
- Vector write operations. Describe how the vector store inserts chunks together with their corresponding embeddings and scopes each stored chunk by index_name and document_id. Explain why the backend generates chunk IDs and why insert returns the number of chunks inserted rather than the chunks themselves.
- Ranked vector search. Explain how search compares a workflow's query vector with vectors in a specified index and returns the top-k most similar chunks ranked by score. Describe how metadata_filters limits eligible chunks and how score_threshold removes low-similarity results even when they would otherwise appear in the top-k.
- Vector store categories. Distinguish database extensions for vector operations from purpose-built vector databases for similarity search. Describe that extensions add vector capabilities to an existing database, whereas purpose-built databases use storage engines and query planners designed for high-dimensional similarity search.
- pgvector operational fit. Explain what pgvector adds to PostgreSQL and why it is a practical choice for teams already running PostgreSQL. State that it provides vector column types, similarity operators, and indexing, while allowing vectors to share existing database infrastructure, backups, and monitoring; with proper indexing and memory configuration it performs well for collections up to roughly 10 million vectors.
- Replaceable backend abstraction. Explain why the section implements pgvector while keeping the backend replaceable through a VectorStore subclass. State that the team can later implement and swap in a new VectorStore backend if it outgrows pgvector, without changing the Data Service.
- pgvector chunk schema. Describe how a single PostgreSQL table stores each chunk's text, embedding, metadata, document ID, and index name. Explain why embedding dimensions must match the configured embedding model and why re-embedding is required after changing models.
- Vector index tradeoffs. Distinguish IVFFlat from HNSW as pgvector indexes for avoiding exhaustive vector comparisons. Explain their different construction requirements and tradeoffs among speed, result quality, memory use, and build time.
- Filtered similarity search. Trace how the pgvector search method executes similarity retrieval in one SQL query: it restricts chunks by index and optional metadata filters, ranks them by cosine similarity, and returns the top-k matches. Explain that pgvector's <=> cosine-distance value is converted to a higher-is-better similarity score by subtracting it from 1.
- Search orchestration flow. Trace how the Data Service search method handles a plain-text query: it retrieves the index configuration, embeds the query, sends the vector plus search options to the vector store, and wraps the returned results.
- Embedding model consistency. Explain why a query must be embedded with the same embedding model that produced an index's chunk vectors, and how the index configuration determines that model.
- Hybrid search approach. Explain that hybrid search runs vector similarity and keyword searches in parallel and merges their results, so semantic matches and exact matches both contribute to a stronger result set.
- Keyword search interface. Describe the keyword_search addition to the VectorStore interface and distinguish it from vector similarity search. Explain that it accepts a raw query string for literal text matching while retaining the same index scope, metadata filters, and result type as search.
- Optional backend keyword capability. Explain why keyword_search has a default implementation that raises NotImplementedError rather than being an abstract method. Describe how this allows vector-only backends to remain minimal while backends such as pgvector or Elasticsearch can override the method to participate in hybrid search.
- Generated tsvector column. Describe how a PostgreSQL tsvector derived from chunk_text enables full-text search, including tokenization, stop-word removal, stemming, and automatic maintenance through GENERATED ALWAYS AS.
- PostgreSQL keyword query path. Trace the PostgreSQL full-text query path: @@ tests a tsvector against a tsquery, and ts_rank gives matching chunks a term-frequency-based score. Justify why this simpler ranking is useful in hybrid search because rank fusion mainly needs the keyword path to retrieve relevant documents in a reasonable order.
- Incompatible search score scales. Explain why vector-search cosine similarity scores and keyword-search rank scores cannot be safely combined directly. Describe why normalizing both scores into a common range can be unreliable.
- Reciprocal Rank Fusion mechanics. Describe how Reciprocal Rank Fusion (RRF) produces a combined ranking from result positions rather than raw search scores. State that each result receives a contribution of 1/(k + rank) from each ranked list, so results near the top of both lists rank highly.
- Hybrid search orchestration. Describe the end-to-end hybrid_search flow: a text query travels simultaneously through vector search and keyword search, and their ranked results are merged with Reciprocal Rank Fusion (RRF). Explain that the caller receives one list ordered by combined relevance.
- Data Service operation categories. Describe how the Data Service gRPC contract groups its capabilities into index management, document ingestion, and search, and relate those categories to the chapter's interfaces and pipelines.
- DataClient translation behavior. Explain how the SDK DataClient translates Python method calls into gRPC requests and converts responses into domain objects, including how search results become a list of SearchResult objects.
- RAG responsibility boundary. Distinguish the infrastructure supplied by the platform for RAG from the retrieval and response decisions owned by a workflow.
What it builds on
- What a session contains (chapter 4) · The distinction between session-based conversation contents and other kinds of information an AI may need.
- Model-managed memory (chapter 4) · The chapter 4 treatment of model-managed memory as a way to retain conversational or user information.
- The Database Schema (chapter 4) · The structured database schema used for session information.
- Providing access to services (chapter 2) · The SDK pattern for providing developers access to platform services.
- The unified provider interface (chapter 3) · The use of a unified interface to hide varying infrastructure implementations behind a platform abstraction.
- The model service contract (chapter 3) · The section assumes the platform can use the Model Service for model-related operations, including the embedding generation later delegated to it.
- The session service contract (chapter 4) · The section refers to integrating retrieval with the Session Service as part of platform workflows.
- The gRPC contract (chapter 3) · The Model Service exposes a gRPC contract that includes embedding operations alongside chat operations.
Section by section
Introduction
This chapter introduces the Data Service as platform infrastructure that grounds AI responses in searchable organizational knowledge rather than relying only on conversational memory.
- Organizational knowledge grounding: Explain why AI applications need grounding in organizational knowledge in addition to conversational memory, and how the Data Service helps responses reflect real company policies, products, and procedures rather than plausible guesses.
Why it matters: Grounding is what lets an assistant answer from your organization’s actual information instead of confidently inventing answers. - Unstructured retrieval problem: Describe why retrieving organizational knowledge differs from looking up conversation records: documents are unstructured, and retrieval must find passages relevant to a natural-language question even when they use different wording.
Why it matters: This difference explains why a dedicated retrieval pipeline and semantic search infrastructure are needed instead of ordinary ID-based data access. - Data Service retrieval pipeline: Trace the high-level Data Service pipeline: it turns documents into meaningful chunks, converts chunks into mathematical representations of meaning, stores them for efficient similarity search, and combines semantic search with metadata filtering.
Why it matters: This pipeline centralizes difficult retrieval infrastructure so individual teams can use searchable knowledge without each building parsing, chunking, embedding, and storage themselves.
Key terms: grounding, Data Service, organizational knowledge, conversational memory, hallucinate, unstructured, semantic matching, natural language, conversation history, metadata filtering
5.1 From documents to searchable knowledge
This section explains how the Data Service turns organizational documents into searchable knowledge for retrieval-augmented AI applications.
- Semantic search rationale: Explain why semantic search is more reliable than keyword matching for connecting a user question to relevant organizational knowledge. Describe how semantic search matches meaning even when the question and document use different words.
Why it matters: Users rarely phrase questions with exactly the same words used in your documents. - Chunk embedding retrieval pipeline: Describe the path from a source document to a search result: parse document text, divide it into semantically coherent chunks, convert chunks and queries into embeddings, and retrieve similar chunk vectors from a vector store. Explain why chunk-level comparison is used instead of comparing a short query with an entire large document.
Why it matters: Retrieval quality depends on preserving meaningful passages and making them comparable to user questions at the right granularity. - Scalable vector similarity search: Explain that embeddings with similar meanings are close in numerical vector space, and that production vector stores use approximate nearest-neighbor algorithms to retrieve similar vectors quickly at large scale. Distinguish this from comparing every query vector with every stored vector.
Why it matters: Fast similarity retrieval makes semantic search practical when an organization has millions of knowledge chunks. - Metadata-aware search results: Describe why retrieval returns more than raw chunk text: it includes the source document, a match-confidence score, and metadata. Explain how metadata filtering restricts semantically similar results to the appropriate domain, such as customer-facing rather than internal content.
Why it matters: Similarity alone cannot ensure that retrieved knowledge is appropriate for the workflow or audience receiving it. - Data Service platform role: State how the Data Service brings together indexes, ingestion, embeddings, vector storage, retrieval, and service integration so workflows can place relevant organizational knowledge into model context for retrieval-augmented generation (RAG).
Why it matters: This shows how document retrieval becomes a reusable platform capability instead of custom logic in every AI workflow.
Key terms: semantic search, keyword matching, similarity matching, ingestion pipeline, chunks, chunking, semantic coherence, embeddings, dense vectors, vector space
5.2 Indexes: organizing knowledge
This section introduces an index as the Data Service unit for organizing related knowledge into isolated, independently configured collections.
- Index as knowledge unit: Explain that an index is the Data Service organizational unit used before document ingestion or search to group related organizational knowledge.
Why it matters: Indexes give the platform a clear place to put and retrieve a particular body of knowledge. - Knowledge isolation boundaries: Describe how indexes keep unrelated knowledge independent, so separate domains such as product documentation, HR policies, and engineering runbooks can use separate collections.
Why it matters: Isolation prevents unrelated organizational knowledge from being treated as one undifferentiated collection. - Per-index retrieval configuration: State that an index is named and has its own embedding configuration and search behavior, rather than every collection necessarily using identical retrieval settings.
Why it matters: Per-index configuration lets the platform define retrieval behavior for a particular body of knowledge.
Key terms: index, Data Service, documents, search, isolated collection, independent body of knowledge, named, embedding configuration, search behavior
5.2.1 Why isolation matters
This subsection explains why separate indexes isolate teams’ knowledge and allow embedding and chunking choices to fit each collection safely.
- Index isolation: Explain how separate indexes keep each team’s searches confined to the relevant document collection, such as Support searching troubleshooting guides and Legal searching compliance documents.
Why it matters: Isolation prevents a workflow from retrieving irrelevant knowledge owned by another team. - Per-index embedding configuration: Describe why embedding-model configuration belongs to an individual index rather than being global: different content domains can use models suited to their material, while the platform can allow only approved models at index creation.
Why it matters: Per-index model choices improve retrieval fit while allowing the platform to prevent unapproved cost or data-compliance risks before ingestion. - Per-index chunking strategies: Explain why chunking strategy is chosen per index: short self-contained FAQ entries should remain intact, whereas long technical manuals need aggressive, section-aware chunking into retrievable pieces.
Why it matters: Chunk sizes and boundaries determine whether retrieved content preserves the useful meaning of the source material.
Key terms: index, index isolation, Data Service, embedding model, embedding configuration, index-level setting, allowed list, index creation, chunking strategy, FAQ entries
5.2.2 Index configuration
This section defines an index’s configuration and explains why embedding settings are immutable, how to migrate models safely, and how chunking and metadata schemas protect retrieval quality.
- Index embedding configuration: Describe the index fields that identify an index and determine how its text becomes vectors, including name, embedding_model, and embedding_dimensions. Explain why the embedding model and dimensions are fixed when the index is created.
Why it matters: Stable embedding configuration prevents incompatible vectors from undermining search quality. - Embedding model migration: Trace the safe process for changing embedding models: create a new index, re-ingest documents, validate retrieval with queries, shift traffic after validation, and retain the old index for rollback. Explain how separate indexes let different embedding models coexist without interference.
Why it matters: A staged index migration changes models without corrupting retrieval or losing a rollback path. - Chunking and metadata controls: Describe how chunking_strategy, chunk_size, and chunk_overlap configure the breaking of ingested documents into chunks. Explain that an optional metadata_schema constrains allowed ingestion metadata and catches inconsistent keys, such as dept versus department, before those inconsistencies cause filtered searches to miss documents.
Why it matters: Consistent chunking and metadata make documents retrievable and keep later metadata filters trustworthy.
Key terms: name, embedding_model, embedding_dimensions, index, re-ingest, validation queries, rollback, index abstraction, embedding configuration, chunking_strategy
5.2.3 Index operations
This section defines index lifecycle operations, the metadata returned for indexes, and why creation, ingestion, naming, and ownership are treated as separate operational concerns.
- Index lifecycle operations: Describe the three index lifecycle operations: creating, listing, and deleting indexes, and explain that they are exposed through an abstract management interface.
Why it matters: A platform needs predictable operations to manage knowledge containers throughout their lifecycle. - Index status metadata: Explain what the Index object reports, including its configuration, document count, total chunks, creation time, and last ingestion time. Describe how last_ingested_at helps teams verify that scheduled or update-triggered ingestion is keeping an index current.
Why it matters: Operational metadata lets teams check the health and freshness of organizational knowledge without inspecting every document. - Creation-ingestion separation: Explain that creating an index sets up an empty container with a name, embedding configuration, chunking strategy, and storage structures, but adds no documents. Distinguish this one-time setup from ongoing ingestion, which supplies documents through the ingestion pipeline on a schedule or after updates.
Why it matters: Separating setup from ongoing document processing makes the platform easier to operate and keeps index configuration distinct from changing content. - Index ownership foundation: Describe why index ownership metadata is needed in a multi-team platform: it provides the basis for checking a caller's identity against the index owner before destructive or write operations. State that the section provides the owner field now, while actual enforcement awaits an authentication mechanism.
Why it matters: Ownership prevents one team or an unauthorized ingestion job from changing another team's knowledge store.
Key terms: create, list, delete, index management operations, Index, document count, total chunks, created_at, last_ingested_at, index creation
5.3 Ingestion pipeline: from raw files to vectors
This section introduces the ingestion pipeline that turns source documents into chunk embeddings stored for semantic search within an index.
- Document ingestion pipeline: Describe the end-to-end ingestion pipeline: it takes a source file, extracts clean text, splits that text into chunks, generates embeddings for the chunks, and stores the results in a vector store.
Why it matters: This pipeline is how organizational documents become knowledge an AI application can retrieve semantically.
Key terms: ingestion pipeline, source file, text extraction, chunks, embeddings, vector store
5.3.1 The challenge of diverse formats
This subsection explains why a platform ingestion pipeline must reliably extract text from many document formats while remaining reusable and extensible for unusual sources.
- Diverse format extraction challenges: Describe why extracting clean, structured text from organizational documents is difficult across PDFs, Word documents, HTML pages, and Markdown. Explain that source formats include presentation or surrounding material that can distort, obscure, or pollute the actual document content.
Why it matters: Retrieval quality depends on turning real-world documents into usable text rather than blindly treating every file as clean content. - Shared platform parsing: Explain the rationale for a shared, platform-level ingestion pipeline rather than separate parsers in every RAG application team. State that the platform solves common parsing problems once, so teams avoid duplicated work and parser fixes benefit every index that uses them.
Why it matters: Centralizing common ingestion work lets application teams focus on their domain while improvements propagate across the organization. - Custom parser extension points: Explain why the ingestion pipeline should provide custom-parser extension points alongside default parsers. State that teams with unusual formatting or domain-specific structures can supply specialized parsing while continuing to use the platform's chunking, embedding, and storage infrastructure.
Why it matters: Extensibility handles exceptional documents without forcing teams to rebuild the rest of the ingestion system.
Key terms: ingestion pipeline, clean, structured text, PDFs, Markdown, HTML pages, Word documents, shared ingestion pipeline, platform-level ingestion pipeline, RAG application, parsers
5.3.2 Pipeline architecture
This section traces how a staged ingestion pipeline converts a raw document into embedded, metadata-bearing chunks that can be searched.
- Document ingestion pipeline: Trace the end-to-end ingestion pipeline from a source file through format detection, parsing, metadata capture, chunking, embedding, and vector-store storage. Explain that each stage has a distinct responsibility and passes its output to the next stage.
Why it matters: A clear pipeline separates document-processing concerns and produces the searchable knowledge that retrieval depends on. - Structure-preserving text extraction: Describe why the parser extracts clean text while preserving headings, paragraphs, and section boundaries. Explain that this structural information is input to the later structure-aware chunking strategy.
Why it matters: Preserved document structure lets chunking respect meaningful content boundaries instead of treating extracted text as an unstructured stream. - Chunk-level metadata propagation: Explain the two sources of metadata captured during ingestion and how it travels with every derived chunk. Distinguish document-derived metadata such as filename, page count, word count, and detected language from caller-supplied metadata such as department, document type, author, version, and tags, and state that it enables filtered search.
Why it matters: Metadata allows retrieval to narrow results to the organizational context a query needs.
Key terms: ingestion pipeline, format detection, parser, text extraction, metadata, chunks, clean text, structural information, headings, paragraphs
5.3.3 Format detection and text extraction
This section explains how the ingestion pipeline detects document formats, extracts structured text through format-specific parsers, and remains extensible through custom parser registration.
- Structured extraction model: Describe how a DocumentParser produces an ExtractedDocument containing sections, headings, hierarchy, and text derived from those sections. Explain how this structure supports both structure-aware chunking and fixed-size chunking.
Why it matters: Preserving document structure lets later retrieval steps chunk content in ways that retain useful context. - Format detection and routing: Explain that the pipeline identifies a file's real format by inspecting file headers first and falling back to extensions when a format lacks a reliable magic-byte signature, then routes the file to the appropriate parser.
Why it matters: Correct routing ensures that each file is interpreted by extraction logic designed for its actual format. - Format-specific parser implementations: Describe why each supported format has its own DocumentParser implementation while conforming to the same parse method and ExtractedDocument output. Give examples of format-specific extraction concerns such as PDF layout, Markdown syntax, or HTML boilerplate.
Why it matters: A common interface allows the pipeline to handle diverse file formats without making downstream stages format-specific. - Custom parser extensibility: Explain how a team adds a new format by implementing a DocumentParser subclass and registering it in the parser registry through the SDK, leaving chunking, embedding, storage, and search unchanged. State why tracking the parser version used for ingestion identifies documents that may need re-ingestion after parser logic changes.
Why it matters: Custom parser registration lets teams support organization-specific sources without modifying pipeline internals or losing traceability when extraction changes. - Text-first multimodal extension: Distinguish the text-first extension approach from true multimodal RAG. Explain that OCR or speech-to-text parsers can convert scanned documents or audio into text for the existing pipeline, whereas direct image-to-text retrieval requires multimodal embeddings in a shared vector space and different vector-storage considerations.
Why it matters: This distinction clarifies what the current text-based architecture can extend easily and what requires a more ambitious retrieval design.
Key terms: DocumentParser, ExtractedDocument, sections, structure-aware chunking, fixed-size chunking, format detection, file headers, file extensions, magic byte signatures, parser routing
5.3.4 Metadata: the filtering foundation
This section explains how document metadata is captured, standardized, and used to filter, attribute, rank, and audit retrieved organizational knowledge.
- Document metadata sources: Describe document metadata as attributes beyond text content, distinguishing metadata extracted during parsing from metadata supplied by the caller at ingestion.
Why it matters: Metadata gives the platform information needed to manage and retrieve documents appropriately, not just match their text. - Audience-based metadata filtering: Explain how metadata filtering narrows vector-search results using attributes such as audience, so a workflow can retrieve customer material without exposing an internal engineering document that discusses the same procedure.
Why it matters: Semantic similarity alone cannot enforce the organizational boundaries that determine which documents a user or workflow should see. - Metadata operational uses: Explain how metadata supports source attribution, freshness-based ranking, and auditing the contents of an index in addition to filtering retrieval results.
Why it matters: These uses make retrieved answers traceable, keep newer material more prominent, and let teams understand their indexed knowledge without inspecting every chunk. - Metadata consistency conventions: Describe why custom_metadata is a flat dictionary of string key-value pairs and why teams should establish shared naming conventions, optionally enforced through metadata_schema validation at ingestion.
Why it matters: Consistent, portable metadata prevents filters from silently missing documents because different teams used incompatible keys or values.
Key terms: metadata, automatic metadata, caller-supplied metadata, ingestion timestamp, custom_metadata, metadata filtering, vector search, audience, customer, internal engineering runbook
5.3.5 Chunking: breaking text into retrievable pieces
This section explains how the Data Service splits extracted documents into retrievable chunks, selects chunking strategies, and preserves context for accurate vector retrieval.
- Chunk quality tradeoffs: Explain why chunking divides extracted text into retrievable pieces and how chunk size, context, and boundaries affect retrieval quality. Describe why chunks that are too large, too small, or split in the middle of a sentence produce poor results.
Why it matters: Chunk boundaries determine whether retrieval returns focused, understandable evidence for an AI response. - Chunk representation and strategy interface: Describe the information a Chunk carries and how the common ChunkingStrategy interface makes chunking pluggable. Explain that index-configured size and overlap are supplied to a strategy, which converts an ExtractedDocument into a list of Chunk objects without requiring pipeline changes.
Why it matters: A stable chunk contract lets the platform change chunking behavior while retaining source traceability and a consistent ingestion pipeline. - Built-in chunking strategies: Distinguish fixed-size chunking, recursive splitting, and structure-aware chunking, including when each is appropriate. Explain why recursive splitting is the platform default and how overlap or document structure preserves meaningful context.
Why it matters: Selecting a strategy that matches the content type is a direct way to improve retrieval accuracy. - Advanced context-preserving approaches: Describe the newer chunking-related approaches of semantic chunking, contextual retrieval, and late chunking, including the extra ingestion cost they trade for improved contextual retrieval. Distinguish whether each approach finds topic boundaries, adds explanatory context, or embeds the full document before creating chunk vectors.
Why it matters: These approaches offer custom options when standard splitting cannot preserve enough context for difficult documents.
Key terms: chunking, chunk size, chunk overlap, retrieval quality, Chunk, text, heading, start_offset, end_offset, metadata
5.3.6 Generating embeddings
This section explains how the Data Service turns text chunks into consistently comparable embeddings through the Model Service, balancing retrieval quality, performance, and operational concerns.
- Index embedding model consistency: Explain why every chunk in an index, and every query searched against it, must use the same embedding model. Describe how index-level configuration enforces this consistency while allowing teams to choose the model appropriate for their content.
Why it matters: Comparable vectors are the foundation of meaningful similarity search and retrieval. - Embedding dimension tradeoffs: Describe embedding dimensionality as the number of floating-point values in a vector, and explain its tradeoff between capturing nuanced semantic relationships and the storage and similarity-computation costs of larger vectors.
Why it matters: Dimension choices affect retrieval quality, storage cost, and search latency for every document in an index. - Model Service embedding delegation: Explain why the Data Service generates embeddings through the Model Service rather than calling embedding providers directly. Distinguish embedding generation, which takes text and returns vectors, from chat completion, which takes messages and returns generated text, while identifying the shared platform infrastructure concerns.
Why it matters: Centralizing provider access keeps ingestion independent of vendors while preserving common reliability, routing, and cost controls. - Batch embedding requests: Describe batching as grouping multiple chunks into one embedding-provider request instead of making one request per chunk. State that 100 is a reasonable default batch size, while the best size depends on provider rate limits and average chunk length.
Why it matters: Batching makes large document-ingestion workloads substantially more efficient.
Key terms: embedding model, index configuration, chunks, queries, vectors, mathematical spaces, dimensionality, floating-point values, semantic relationships, storage costs
5.3.7 Document lifecycle
This section explains how the data service safely updates documents through replace-on-reingest, atomic ingestion, and index-level swaps for consistent bulk updates.
- Replace-on-reingest strategy: Explain how replace-on-reingest updates a document when it is ingested again with the same identifier, and justify why the platform favors it over diff-based updates.
Why it matters: A simple replacement strategy keeps searchable knowledge aligned with the newest document version without fragile version-diff logic. - Ingestion entry point and identifiers: Describe the ingestion method as the single entry point that orchestrates format detection, text extraction, chunking, embedding generation, and storage, including the role of an optional stable document_id.
Why it matters: One ingestion entry point lets callers submit a file and metadata while the service consistently runs the complete indexing pipeline. - Atomic and bulk re-ingestion safety: Explain why replace-on-reingest must be atomic for one document, and describe the new-index-and-traffic-swap approach when a bulk re-ingestion cannot tolerate a mixed index state.
Why it matters: These safeguards prevent failed updates from removing searchable knowledge and prevent users from seeing inconsistent results during large migrations.
Key terms: replace-on-reingest, diff-based updates, chunks, vectors, re-embedding, ingestion method, document_id, format detection, text extraction, chunking
5.3.8 Document management
This section defines Data Service operations for listing, inspecting, and safely deleting indexed documents without exposing the vector store directly.
- Document management operations: Describe the three Data Service document management operations and what each returns or removes. Explain that list_documents shows metadata for every document in an index, get_document retrieves one document's metadata by ID, and delete_document removes a document and its indexed content.
Why it matters: These operations let applications and platform teams inspect, verify, and maintain the knowledge stored in an index. - Atomic cascading deletion: Explain why delete_document performs an atomic cascading delete across the document record, its chunks, and corresponding vector-store vectors. Describe how partial deletion can leave orphaned vectors that silently corrupt search results.
Why it matters: Keeping related indexed data consistent prevents deleted or outdated content from continuing to influence retrieval.
Key terms: list_documents, get_document, delete_document, source URI, ingestion timestamp, chunk count, cascading delete, atomic, document record, chunks
5.3.9 Asynchronous ingestion
This section introduces asynchronous ingestion so large or batch document uploads return immediately and are processed in the background with trackable job status.
- Asynchronous ingestion rationale: Explain why synchronous ingestion becomes impractical for very large documents or batches, and why the platform should return immediately while parsing, chunking, embedding, and storage run in the background.
Why it matters: Background processing prevents long-running ingestion work from blocking the workflow that submitted it. - Ingestion job tracking: Describe the asynchronous ingestion job interface: submitting a file returns a job ID immediately, and teams poll the job to observe its state, progress, successful document ID, or failure error.
Why it matters: Job tracking lets teams submit many documents quickly while retaining visibility and actionable failure information for long-running work.
Key terms: synchronous ingestion, asynchronous ingestion, background, ingestion pipeline, job_id, queued, processing, completed, failed, progress
5.4 Vector storage and search
This section introduces vector storage as a backend-independent abstraction for storing embedding vectors and searching them by similarity.
- Vector storage abstraction: Explain why the Data Service uses a consistent vector-store abstraction instead of tying ingestion and retrieval to one vector database backend. Describe how this lets teams select a backend for their scale and performance needs while the rest of the Data Service continues to work the same way.
Why it matters: A backend-independent interface lets the platform evolve its storage choices without rewriting the surrounding Data Service. - Vector and session storage contrast: Distinguish vector storage from session storage in terms of the data and lookup operations involved. State that session storage handles sequential messages retrieved by ID, whereas vector storage handles floating-point embedding arrays retrieved through similarity search, even though both use the same abstraction pattern.
Why it matters: Recognizing the different access patterns prevents applying session-storage assumptions to knowledge retrieval.
Key terms: vector store, storage abstraction, vector database backend, consistent interface, embeddings, similarity search, session storage, vector storage, sequential messages, ID
5.4.1 Vector store interface
This section defines a backend-agnostic vector store interface for storing embedded chunks, deleting scoped content, and retrieving ranked, filtered search results.
- Vector write operations: Describe how the vector store inserts chunks together with their corresponding embeddings and scopes each stored chunk by index_name and document_id. Explain why the backend generates chunk IDs and why insert returns the number of chunks inserted rather than the chunks themselves.
Why it matters: A consistent write interface lets the ingestion pipeline store knowledge without depending on a particular vector database. - Scoped deletion safeguards: Explain how delete_by_document removes all chunks for one document and delete_index removes all chunks in an index, using the stored document_id and index_name scopes. Describe why both methods return deletion counts and why production systems should authorize destructive deletes against the index owner.
Why it matters: Scoped, authorized deletion supports safe re-ingestion and prevents an accidental call from destroying a shared knowledge base. - Ranked vector search: Explain how search compares a workflow's query vector with vectors in a specified index and returns the top-k most similar chunks ranked by score. Describe how metadata_filters limits eligible chunks and how score_threshold removes low-similarity results even when they would otherwise appear in the top-k.
Why it matters: Retrieval must select relevant context while excluding weak matches that consume tokens and can confuse the model. - Search result contract: State what each SearchResult provides: chunk text, a similarity score from 0 to 1, its document ID, and attached metadata. Explain how a workflow can use these fields for model context, relevance decisions, and source citations while remaining independent of backend indexing and similarity details.
Why it matters: A standard result shape lets workflows use retrieved evidence consistently across interchangeable vector-store backends.
Key terms: VectorStore, insert, chunks, embeddings, index_name, document_id, delete_by_document, delete_index, owner, search
5.4.2 Choosing a vector store backend
This section compares vector-store backend options and explains why the platform implements pgvector behind a replaceable VectorStore abstraction.
- Vector store categories: Distinguish database extensions for vector operations from purpose-built vector databases for similarity search. Describe that extensions add vector capabilities to an existing database, whereas purpose-built databases use storage engines and query planners designed for high-dimensional similarity search.
Why it matters: This distinction helps you evaluate whether to extend infrastructure you already operate or adopt a specialized vector system. - pgvector operational fit: Explain what pgvector adds to PostgreSQL and why it is a practical choice for teams already running PostgreSQL. State that it provides vector column types, similarity operators, and indexing, while allowing vectors to share existing database infrastructure, backups, and monitoring; with proper indexing and memory configuration it performs well for collections up to roughly 10 million vectors.
Why it matters: Using pgvector can provide capable vector search without introducing a separate system for a team that already operates PostgreSQL. - Specialized backend tradeoffs: Describe the scaling and operational tradeoffs of purpose-built vector databases. Explain that they generally scale further before performance degrades, but self-hosted options add deployment and monitoring work, while managed services such as Pinecone reduce operational effort in exchange for possible vendor lock-in and usage-based costs that can rise at scale.
Why it matters: A backend choice must balance search scale against the operational and commercial costs the organization is willing to accept. - Replaceable backend abstraction: Explain why the section implements pgvector while keeping the backend replaceable through a VectorStore subclass. State that the team can later implement and swap in a new VectorStore backend if it outgrows pgvector, without changing the Data Service.
Why it matters: A stable backend abstraction lets the platform start with familiar infrastructure without locking the Data Service to that choice.
Key terms: vector store, database extensions, purpose-built vector databases, similarity search, high-dimensional similarity search, pgvector, PostgreSQL, vector column types, similarity operators, indexing
5.4.3 The pgvector implementation
This section implements the VectorStore interface with PostgreSQL and pgvector, covering chunk storage, vector indexing, and filtered similarity search.
- pgvector chunk schema: Describe how a single PostgreSQL table stores each chunk's text, embedding, metadata, document ID, and index name. Explain why embedding dimensions must match the configured embedding model and why re-embedding is required after changing models.
Why it matters: The schema preserves retrievable text and its organizational context while making vector search possible. - Vector index tradeoffs: Distinguish IVFFlat from HNSW as pgvector indexes for avoiding exhaustive vector comparisons. Explain their different construction requirements and tradeoffs among speed, result quality, memory use, and build time.
Why it matters: Choosing an index determines whether vector search remains practical as a collection grows while retaining enough retrieval quality. - Filtered similarity search: Trace how the pgvector search method executes similarity retrieval in one SQL query: it restricts chunks by index and optional metadata filters, ranks them by cosine similarity, and returns the top-k matches. Explain that pgvector's <=> cosine-distance value is converted to a higher-is-better similarity score by subtracting it from 1.
Why it matters: Database-side filtered ranking returns only relevant, usable chunks without moving all embeddings into application code. - Document lifecycle operations: Describe how document and index identifiers support lifecycle management in the pgvector table. State that re-ingestion deletes rows for the matching index name and document ID before inserting replacement chunks, and that deleting an index removes its associated rows.
Why it matters: Lifecycle identifiers prevent stale chunks from remaining searchable after documents or indexes change.
Key terms: pgvector, vector column, embedding column, chunk_text, JSONB, document_id, IVFFlat, Inverted File Flat, HNSW, Hierarchical Navigable Small World
5.4.4 Search orchestration
This section shows how the Data Service turns a plain-text query into vector-store search results by coordinating index configuration, embedding generation, and vector search.
- Search orchestration flow: Trace how the Data Service search method handles a plain-text query: it retrieves the index configuration, embeds the query, sends the vector plus search options to the vector store, and wraps the returned results.
Why it matters: This orchestration is the path that lets application workflows search organizational knowledge without handling embeddings or vector stores themselves. - Embedding model consistency: Explain why a query must be embedded with the same embedding model that produced an index's chunk vectors, and how the index configuration determines that model.
Why it matters: Using the index's embedding model keeps query and document vectors compatible for meaningful similarity search. - Vector search limitations: Distinguish the semantic natural-language queries that vector similarity handles well from exact matches, keywords, and identifiers that it struggles with, motivating hybrid search.
Why it matters: Recognizing this limitation explains why the platform adds keyword matching instead of relying only on vector similarity.
Key terms: Data Service, search method, index configuration, embedding generator, vector store, filters, embedding model, query, chunk vectors, vector similarity
5.5 Hybrid search: combining vectors with keywords
This section explains why hybrid search combines semantic vector search with keyword search to cover both meaning-based and exact-match queries.
- Semantic search tradeoffs: Describe how semantic search uses embeddings to retrieve content with similar meaning even when the query and document use different words, and explain why it can miss or dilute precise identifiers.
Why it matters: Semantic retrieval is valuable for vocabulary gaps, but it is not reliable by itself when an exact code or citation matters. - Keyword search tradeoffs: Describe why keyword search excels at literal identifiers, partial matches, boolean queries, and phrase matching, while failing when relevant content uses different vocabulary from the query.
Why it matters: Keyword retrieval provides the precision needed for exact terms, but its dependence on overlapping words makes it insufficient alone. - Hybrid search approach: Explain that hybrid search runs vector similarity and keyword searches in parallel and merges their results, so semantic matches and exact matches both contribute to a stronger result set.
Why it matters: Using both retrieval paths prevents the system from having to choose between flexible meaning matching and precise literal matching.
Key terms: semantic search, embeddings, dense vectors, vector similarity search, keyword search, traditional text search, exact matches, partial matches, boolean queries, phrase matching
5.5.1 Adding keyword search to the platform
This section extends the VectorStore interface with an optional keyword retrieval method so the platform can support hybrid search without disrupting vector-only backends.
- Keyword search interface: Describe the keyword_search addition to the VectorStore interface and distinguish it from vector similarity search. Explain that it accepts a raw query string for literal text matching while retaining the same index scope, metadata filters, and result type as search.
Why it matters: A consistent retrieval interface lets later orchestration combine keyword and vector results without handling incompatible response shapes. - Optional backend keyword capability: Explain why keyword_search has a default implementation that raises NotImplementedError rather than being an abstract method. Describe how this allows vector-only backends to remain minimal while backends such as pgvector or Elasticsearch can override the method to participate in hybrid search.
Why it matters: Optional capabilities prevent every storage backend from being forced to implement features it cannot or does not need to support. - Vector database naming convention: State why the interface continues to be called VectorStore even after gaining keyword search support. Explain that the name follows industry convention, where vector databases can also offer keyword or hybrid retrieval.
Why it matters: This naming context helps readers avoid assuming that a vector-oriented store can only perform vector similarity retrieval.
Key terms: VectorStore, keyword_search, search, raw query string, embedding vector, index scoping, NotImplementedError, abstract method, backend, FAISS
5.5.2 PostgreSQL keyword search implementation
This section implements PostgreSQL keyword search over document chunks using generated tsvectors, while explaining how its ranking differs from BM25 and why it is sufficient for hybrid retrieval.
- BM25 ranking factors: Explain how BM25 ranks keyword-search results using term frequency, inverse document frequency, and document length, and distinguish it from PostgreSQL's built-in ranking.
Why it matters: This lets you judge what ranking quality PostgreSQL full-text search provides compared with a dedicated search engine. - Generated tsvector column: Describe how a PostgreSQL tsvector derived from chunk_text enables full-text search, including tokenization, stop-word removal, stemming, and automatic maintenance through GENERATED ALWAYS AS.
Why it matters: A generated search vector adds keyword retrieval to existing chunks without requiring the ingestion pipeline to write separate search-indexing code. - PostgreSQL keyword query path: Trace the PostgreSQL full-text query path: @@ tests a tsvector against a tsquery, and ts_rank gives matching chunks a term-frequency-based score. Justify why this simpler ranking is useful in hybrid search because rank fusion mainly needs the keyword path to retrieve relevant documents in a reasonable order.
Why it matters: Hybrid retrieval benefits when keyword search reliably surfaces relevant chunks even if its standalone ranking is not a perfect BM25 implementation.
Key terms: BM25, Best Matching 25, term frequency, inverse document frequency, document length, Elasticsearch, tsvector, chunk_text, GENERATED ALWAYS AS, stop words
5.5.3 Merging results: Reciprocal Rank Fusion
This section explains how Reciprocal Rank Fusion combines vector and keyword search rankings without comparing their incompatible raw scores, and why it is usually the practical hybrid-search default.
- Incompatible search score scales: Explain why vector-search cosine similarity scores and keyword-search rank scores cannot be safely combined directly. Describe why normalizing both scores into a common range can be unreliable.
Why it matters: A hybrid search system needs a merging method that does not mistake differently scaled scores for equivalent relevance. - Reciprocal Rank Fusion mechanics: Describe how Reciprocal Rank Fusion (RRF) produces a combined ranking from result positions rather than raw search scores. State that each result receives a contribution of 1/(k + rank) from each ranked list, so results near the top of both lists rank highly.
Why it matters: RRF lets hybrid search preserve useful signals from both semantic and keyword retrieval without requiring compatible scoring systems. - RRF constant and practical tradeoffs: State the role of the RRF constant k and identify k=60 as the standard empirical default described here. Distinguish RRF from fragile weighted-score merging and from learned ranking models that can perform best with sufficient training data but require a separate ranking ML pipeline.
Why it matters: Understanding these tradeoffs helps teams choose a robust default before taking on score tuning or a dedicated learning-to-rank system.
Key terms: cosine similarity, keyword search, score distributions, normalization, heterogeneous search results, Reciprocal Rank Fusion, RRF, rank, 1/(k + rank), combined score
5.5.4 Putting it together
This section combines vector and keyword retrieval into one hybrid-search operation that fuses parallel ranked results into a consistent response.
- Hybrid search orchestration: Describe the end-to-end hybrid_search flow: a text query travels simultaneously through vector search and keyword search, and their ranked results are merged with Reciprocal Rank Fusion (RRF). Explain that the caller receives one list ordered by combined relevance.
Why it matters: Hybrid search lets the Data Service use semantic similarity and exact keyword relevance together without exposing two retrieval paths to callers. - Candidate oversampling strategy: Explain why each search path retrieves more candidates than the caller requested before rank fusion, such as fetching ten candidates from each path for a request of five. Describe how this broader pool lets RRF reward chunks that rank moderately in both paths but could otherwise be excluded.
Why it matters: Oversampling improves the chance that fusion can identify results supported by both semantic and keyword evidence. - Consistent search result interface: Describe how keyword_search mirrors vector search in its output contract: both query the chunks table, apply the same metadata filters, and return List[SearchResult], while differing only in matching logic. Explain that this lets callers switch between hybrid or vector search without changing result consumption.
Why it matters: A shared result contract keeps retrieval implementations interchangeable for application code.
Key terms: hybrid_search, vector search, keyword search, Model Service, cosine similarity, tsquery, oversampling, candidates, RRF, recall
5.6 Service contract and complete retrieval flow
This section exposes the Data Service through a gRPC contract and thin SDK client, clarifying how retrieval capabilities compose into RAG while workflows retain application-level decisions.
- Data Service operation categories: Describe how the Data Service gRPC contract groups its capabilities into index management, document ingestion, and search, and relate those categories to the chapter's interfaces and pipelines.
Why it matters: This contract gives workflows one consistent way to manage organizational knowledge and retrieve it. - DataClient translation behavior: Explain how the SDK DataClient translates Python method calls into gRPC requests and converts responses into domain objects, including how search results become a list of SearchResult objects.
Why it matters: The thin client lets workflow authors use ordinary Python objects instead of directly handling protobuf RPC messages. - RAG responsibility boundary: Distinguish the infrastructure supplied by the platform for RAG from the retrieval and response decisions owned by a workflow.
Why it matters: Keeping platform capabilities separate from application logic makes reusable infrastructure possible without forcing every application to retrieve or respond the same way.
Key terms: Data Service, gRPC contract, index management, document ingestion, search, hybrid search, DataClient, gRPC, RPC, stub
5.7 Summary
This summary explains how the Data Service turns organizational documents into independently managed, searchable knowledge that workflows retrieve to ground AI responses.
Where this is used later
- Grounding Claw in reality: knowledge retrieval and agentic RAG (chapter 10) · Grounding Claw in reality uses knowledge retrieval and agentic RAG.
- Domain-specific telemetry (chapter 7) · Domain-specific telemetry can report service-level information about indexed knowledge and retrieval.
- Output guardrails: the last line of defense (chapter 6) · The retrieval flow supplies organizational data that can serve as authoritative sources for checking claims.
- Offline evaluation: comparing variants before deployment (chapter 8) · The complete retrieval flow that can be incorporated into an end-to-end RAG evaluation.
- (chapter intro) (chapter 10) · The Data Service retrieval flow.
- The prompt that makes it all work: context engineering (chapter 10) · Context engineering depends on workflow-level choices about how retrieved context is formatted and included in a prompt.
Concepts explained in this chapter
- chunkingFixed-size chunking divides text into chunks containing a configured number of tokens with optional overlap between consecutive chunks.
- retrieval indexA retrieval index is a searchable data structure that organizes document representations and associated metadata so relevant information can be found efficiently.
- text chunkA document chunk is a segment of an ingested document, with associated source text or metadata, stored as an independently searchable unit.
- vector databaseA vector database is a database system optimized for storing, indexing, and retrieving vector representations.
- hybrid searchHybrid retrieval combines multiple retrieval methods, such as semantic vector search and keyword search, into a unified result ranking.
- metadata filterA metadata filter is a query condition that restricts candidate records according to descriptive key-value attributes.
- data ingestionDocument ingestion is the process of importing a document, extracting and chunking its content, generating representations, and storing the resulting data for retrieval.
- vectorizationAn embedding model is an AI model that converts content into numerical vector representations capturing semantic or learned relationships.
- similarity searchVector search retrieves items by comparing numerical vector representations of queries and candidate content.
- index configurationAn index configuration is the set of immutable and adjustable parameters that define an index's identity, embedding model, and document chunking behavior.
- embeddingAn embedding is a numerical vector representation of content that preserves aspects of its meaning so semantically similar content has nearby representations.
- pgvectorpgvector is a PostgreSQL extension that provides vector data types, similarity operators, and indexes for storing and querying embeddings.
- parsingParsing is the analysis of structured or semi-structured input to identify and extract its meaningful components.
- lexical searchKeyword search retrieves content by matching literal terms or tokens in a query against indexed content.
- structure-aware chunkingStructure-aware chunking is the division of document text into chunks while preserving or using structural boundaries such as headings and sections.
- RecursiveCharacterTextSplitterRecursive splitting divides text by trying a hierarchy of natural separators and recursively using finer separators when pieces exceed the configured chunk size.
- similarity metricA similarity score is a numerical value representing how closely two items or vector representations match, with larger values indicating greater similarity.
- reciprocal rank fusionReciprocal rank fusion is a rank-aggregation algorithm that combines result lists using the reciprocal positions of items rather than their scores.
- schemaA database schema is the formal organization of a database's tables, columns, relationships, and constraints.
- ivfflatIVFFlat is an approximate vector index that partitions vectors into clusters and searches selected clusters to accelerate similarity queries.
- data serviceA data service is a reusable platform service that ingests documents, generates vector embeddings, and supports semantic search over organizational knowledge.
- HNSWHNSW is a graph-based approximate nearest-neighbor index that organizes vectors into layered networks of nearby connections.
- cosine distanceCosine distance measures dissimilarity between vectors based on the angle between their directions, with zero indicating identical direction.
- dimensionDimensionality is the number of scalar values composing a vector representation.
- format detectionFormat detection is the process of determining the file format of an input document before selecting a parser.
- retrieval rankingA rank is the ordinal position of an item in an ordered list of search results.
- replace-on-reingestReplace-on-reingest updates indexed content by deleting data for an existing document and rebuilding it from the newly ingested version.
- groundingGrounding is the process of connecting an AI application's responses to authoritative external or organizational knowledge.
- organizational knowledgeOrganizational knowledge is the body of policies, procedures, guides, directories, and other information maintained by an organization.
- retrieval-augmented generationA retrieval-augmented generation pipeline retrieves relevant context and supplies it to a generative model when producing a response.
- atomicityAtomicity is the database transaction property that requires all operations in a transaction to succeed together or none to take effect.
- storage interfaceA storage interface is an abstract contract that defines operations for persisting and retrieving application data independently of a backend.
- batchingBatching is the accumulation of multiple items so they can be processed or transmitted together.
- model selectionModel selection is the process of choosing an AI model for a task based on criteria such as quality, cost, latency, or reliability.
- database transactionA database transaction is a sequence of database operations treated as one unit of work with a shared commit or rollback outcome.
- gRPC contractA gRPC contract is a formal service interface that defines the operations and interaction structure exposed between gRPC services.
- isolationIsolation is the separation of software workloads so that their execution, resources, and interactions are constrained from affecting one another.
- job idA job ID is a unique identifier assigned to an asynchronous workflow execution so clients and services can refer to its job record.
- storage abstractionA storage abstraction is an interface or design boundary that separates data-storage operations from the underlying persistence technology.
- nlp-based chunkingSemantic chunking detects topic boundaries by comparing embeddings of consecutive sentences and starting chunks where semantic similarity drops sharply.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
