Document indexing converts raw text into vector representations that support semantic search. In a RAG workflow, it prepares document content for later retrieval. section 7.2.2 section 7.2.4
A typical indexing workflow:
- Collect documents and split them into manageable chunks that preserve context. section 7.2.4
- Convert chunks into dense vectors with an embedding model, then store them in a vector database for semantic search. section 7.2.4
- Attach metadata such as source document, page number, and processing timestamp to support later retrieval. section 7.2.4
Listing 7.8 Indexing documents using a vector store
import { HNSWLib } from '@langchain/community/vectorstores/hnswlib';
async indexDocumentsFromDirectory(documentDirectory) {
const documents = [];
// Find all PDF files in the directory
const pdfFiles = fs
.readdirSync(documentDirectory)
.filter((file) =>
path.extname(file).toLowerCase() === '.pdf')
.map((file) =>
path.join(documentDirectory, file)); #1
// Process each PDF document
for (const filePath of pdfFiles) { #2
const processedDocs = await this.processDocument(filePath);
documents.push(...processedDocs);
}
// Create vector store from processed documents
const vectorStore =
await HNSWLib.fromDocuments(documents, this.embeddings); #3
console.log(`Indexed ${documents.length}
document chunks from ${pdfFiles.length} documents`);
return vectorStore;
}
The JavaScript example gathers PDFs from a directory, processes each document, and creates an HNSWLib vector store from the resulting chunks. section 7.2.4
