Document parsing extracts and structurally interprets content from files such as PDFs and DOCX documents. In document summarization and RAG workflows, parsing prepares source material for cleaning, embedding, retrieval, and grounded conversation. section 7.1.4 section 11.1.1
A practical parsing pipeline accounts for both content quality and operational safety:
- Extract text while allowing for scattered content, encoded PDF text, tables, footnotes, annotations, and embedded images. section 7.1.4
- Normalize parsed content by joining content arrays, trimming whitespace, removing unwanted references, collapsing spaces, and filtering non-content lines. section 7.1.4
- Handle images separately when the PDF loader overlooks them, using image extraction followed by OCR or external language-model descriptions. section 7.1.4
- Validate file types and sizes, process parsing and chunking on the server, and add error handling before vectorizing content. section 11.2.2
Knowledge-base upload flow
A user creates a knowledge base, uploads a PDF or DOCX file, and waits while the application parses and analyzes it. The resulting content becomes vector embeddings stored in Upstash Vector, enabling similarity search and conversational questions grounded in the uploaded documents. section 11.1.1
Listing 7.3 Normalization of document content
function normalizeDocuments(docs) {
return docs.map((doc) => {
let pageContent;
if (typeof doc.pageContent === "string") {
pageContent = doc.pageContent;
} else if (Array.isArray(doc.pageContent)) {
pageContent = doc.pageContent.join("\n"); #1
}
if (pageContent) {
pageContent = pageContent.trim(); #2
// Remove unwanted characters (e.g., figure references)
pageContent = pageContent.replace(/Figure \d+:.*?\n/g, '').replace(/−\
➥d+\n/g, '');
// Normalize spaces
pageContent = pageContent.replace(/\s+/g, ' '); #3
pageContent = pageContent.replace(/(\s*l\s*){2,}/g, ' ');
// Filter out non-content lines
const filteredDocument = pageContent.split('\n').filter(line =>
!line.includes('References') && !line.includes('Acknowledgments')
)[0]; #4
doc.pageContent = filteredDocument;
return doc;
}
return null;
}).filter(doc => doc.content !== null);
// Filter out any null entries from the result
}
Use the normalization listing to inspect a JavaScript cleanup function that joins content, trims it, removes unwanted characters, normalizes spaces, and filters lines. section 7.1.4
