RecursiveCharacterTextSplitter divides loaded documents into manageable, similarly sized chunks before they enter a searchable vector store. This keeps retrieved prompts smaller while preserving context between neighboring chunks. section 10.5.2
Configure the splitter by choosing:
- chunk_size=1000, which targets 1,000 characters per chunk. section 10.5.2
- chunk_overlap=200, which repeats 200 characters between adjacent chunks to maintain context. section 10.5.2
- split_documents(documents), which produces texts ready for storage and embedding. section 10.5.2
Preparing support articles for RAG
Load .txt articles with DirectoryLoader, split them with RecursiveCharacterTextSplitter, and pass the resulting chunks to add_documents. Pinecone then stores the text alongside embeddings so similarity_search can retrieve relevant excerpts for a user's question. section 10.5.2
splitter = RecursiveCharacterTextSplitter(
chunk_size=1000,
chunk_overlap=200
)
texts = splitter.split_documents(documents)
Inspect the minimal splitter configuration and the call that converts loaded documents into searchable chunks. section 10.5.2
