RecursiveCharacterTextSplitter divides loaded document text into smaller character-based chunks for embedding and retrieval workflows. In the MongoDB Atlas pipeline, it runs after PyPDFLoader and before OpenAIEmbeddings. section 14.5.2
Configure the splitter around the embedding workload:
- Set chunk_size=500 to create 500-character chunks. section 14.5.2
- Set chunk_overlap=0 when the ingestion flow requires no repeated characters between chunks. section 14.5.2
- Pass the loaded documents to split_documents before generating embeddings and inserting them into MongoDB Atlas. section 14.5.2
Embedding-ingestion configuration
from langchain.text_splitter import RecursiveCharacterTextSplitter text_splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=0) docs = text_splitter.split_documents(data) section 14.5.2
import os from langchain_community.document_loaders import PyPDFLoader from langchain_openai import OpenAIEmbeddings from langchain.text_splitter import RecursiveCharacterTextSplitter from langchain_community.vectorstores import MongoDBAtlasVectorSearch from pymongo import MongoClient MONGO_URI = os.environ["MONGO_URI"] # Note that if you change this, you also ➥need to change it in `rag_mongo/chain.py` DB_NAME = "langchain" #1 COLLECTION_NAME = "mongodb" #2 ATLAS_VECTOR_SEARCH_INDEX_NAME = "default" #3 EMBEDDING_FIELD_NAME = "embedding" client = MongoClient(MONGO_URI) db = client[DB_NAME] MONGODB_COLLECTION = db[COLLECTION_NAME] if __name__ == "__main__": # Load docs loader = PyPDFLoader("https://mng.bz/PwxP") #4 data = loader.load() # Split docs text_splitter = RecursiveCharacterTextSplitter ➥(chunk_size=500, chunk_overlap=0) docs = text_splitter.split_documents(data) # Insert the documents in MongoDB Atlas Vector Search _ = MongoDBAtlasVectorSearch.from_documents( documents=docs, embedding=OpenAIEmbeddings(disallowed_special=()), collection=MONGODB_COLLECTION, index_name=ATLAS_VECTOR_SEARCH_INDEX_NAME, )
Inspect the complete ingestion listing, including PDF loading, splitting, embeddings, MongoDB connection, and vector-search insertion. section 14.5.2
