Document normalization transforms parsed content into a cleaner, more consistent representation before summarization. It focuses processing on substantive text and reduces unnecessary token use. section 7.1.4
Apply normalization to common extraction problems:
- Remove headers, footers, page numbers, watermarks, figure references, and other unwanted artifacts. section 7.1.4
- Trim content, collapse repeated whitespace, and filter lines such as references or acknowledgments. section 7.1.4
- Consider stopword removal to reduce noise and token consumption, while recognizing that images may require OCR or an external LLM. section 7.1.4
Listing 7.3 Normalization of document content
function normalizeDocuments(docs) {
return docs.map((doc) => {
let pageContent;
if (typeof doc.pageContent === "string") {
pageContent = doc.pageContent;
} else if (Array.isArray(doc.pageContent)) {
pageContent = doc.pageContent.join("\n"); #1
}
if (pageContent) {
pageContent = pageContent.trim(); #2
// Remove unwanted characters (e.g., figure references)
pageContent = pageContent.replace(/Figure \d+:.*?\n/g, '').replace(/−\
➥d+\n/g, '');
// Normalize spaces
pageContent = pageContent.replace(/\s+/g, ' '); #3
pageContent = pageContent.replace(/(\s*l\s*){2,}/g, ' ');
// Filter out non-content lines
const filteredDocument = pageContent.split('\n').filter(line =>
!line.includes('References') && !line.includes('Acknowledgments')
)[0]; #4
doc.pageContent = filteredDocument;
return doc;
}
return null;
}).filter(doc => doc.content !== null);
// Filter out any null entries from the result
}
Inspect the JavaScript normalizeDocuments implementation, which handles string or array content and applies cleanup operations. section 7.1.4
