Data preprocessing turns source data into a form a generative AI application can use. In a retrieval-augmented generation (RAG) pipeline, it can reduce noise, improve the quality of generated answers, and help determine a suitable size for document chunks. section 7.7.2 section 10.2.3
Choose operations according to the data and application:
- Clean unwanted noise and transform data from one format to another. section 7.7.2
- Use information extraction, feature normalization, tokenization, or stop-word removal when required. section 7.7.2
- Chunk prepared content with attention to query complexity, the model’s context window, and any metadata the application needs. section 7.7.2
For enterprise RAG, preprocessing belongs in the data layer alongside connectors and pipelines that ingest records through APIs or plugins. Pipelines may need batch or streaming processing, authentication, data-refresh settings, and scalable, fault-tolerant processing. Prepared data can then support embedding creation and vector-database retrieval. section 10.2.3
Preparing content for different questions
A short factual question such as “What is the capital of the UK?” places different demands on retrieved content than a question about AI’s economic implications across US industries. Clean the source content, then compare chunk sizes against both answer quality and performance rather than assuming one size fits both queries. section 7.7.2
Table 7.3 Chunking approaches
|
Chunking approach
|
Description
|
|---|---|
|
Fixed length
|
Divide documents into chunks of a fixed number of words or tokens. This is straightforward but may sometimes split information that ideally should be kept together.
|
|
Sliding window
|
Use a fixed-sized sliding window with or without overlapping data. This can ensure that important boundaries within the text are not missed, but it can also lead to redundancy if there’s significant overlap.
|
|
Punctuation based
|
Divide the text based on punctuation, such as paragraphs or sections. This is less arbitrary than fixed-length chunking and often preserves the semantic integrity of the content. However, it can result in variable chunk sizes.
|
|
Topic or section breaks
|
In structured documents such as Wikipedia articles, natural breaks like sections or subsections can be used to define chunks. This method ensures that the content within a chunk is semantically coherent.
|
|
Adaptive
|
Use algorithms or models that adaptively determine the best way to chunk documents based on their content. This can be more complex but might yield semantically cohesive chunks.
|
Compare fixed-length, sliding-window, punctuation-based, and section-based chunking before choosing how to divide preprocessed documents. section 7.7.2
Unstructured provides functions and connectors for customized preprocessing pipelines, including conversion of unstructured data into structured output. LangChain and SK offer chunking options with configurable overlap to help preserve context. Treat these as choices to evaluate against the corpus and application requirements. section 10.4.2 section 7.7.2
