A retrieval system can return inaccurate or irrelevant answers when ingestion strips away how content is organized. Document structure preserves relationships among headings, tables, lists, and their surrounding text so search can use that context. section 6.4.3
Preserve structure through the ingestion workflow:
- Connect content stores or migrate content while maintaining data integrity. section 6.4.3
- Extract headings, tables, and lists instead of treating every document as undifferentiated text. section 6.4.3
- Chunk large documents into representative subdocuments for granular indexing and retrieval. section 6.4.3
- Use loaders and transformers to process source formats and retain useful metadata before retrieval. section 6.4.3
Listing 6.2 HTML splitter
# Install langchain-text-splitters if not already installed
%pip install -qU langchain-text-splitters
# Import necessary modules
import langchain
import langchain_text_splitters
print("Langchain version:", langchain.__version__)
print("Langchain Text Splitters module loaded successfully!")
from langchain_text_splitters import HTMLHeaderTextSplitter
from langchain.schema import Document # Ensure Document is properly imported
from bs4 import BeautifulSoup
print("BeautifulSoup is installed successfully!")
# Sample HTML content to be split
html_string = """
<!DOCTYPE html>
<html>
<body>
<div>
<h1>Introduction</h1>
<p>This is the introduction section of the document.</p>
<div>
<h2>Chapter 1: Getting started</h2>
<p>This section covers the basics of getting started.</p>
<h3>Section 1.1 Setup</h3>
<p>This subsection explains the setup process.</p>
<h3>Section 1.2 Configuration</h3>
<p>This subsection details the configuration options.</p>
</div>
<div>
<h2>Chapter 2: Advanced Techniques</h2>
<p>This section dives into more advanced techniques.</p>
</div>
<br/> <!-- Fix: Ensuring self-closing tag is correctly formatted -->
<p>What you learned in the Introduction.</p>
</div>
</body></html>
"""
# Define header tags to split on (h1, h2, h3 represent different levels of headers)
headers_to_split_on = [
("h1", "Header 1"), # Top-level headers
("h2", "Header 2"), # Subsection headers
("h3", "Header 3"), # Sub-subsection headers
]
# Initialize the HTML header text splitter with the specified header levels
html_splitter = HTMLHeaderTextSplitter(headers_to_split_on=headers_to_split_on)
# Split the HTML document into structured chunks
html_header_splits = html_splitter.split_text(html_string)
# Display structured output
for doc in html_header_splits:
print(f"Content:\n{doc.page_content}\nMetadata: {doc.metadata}\n{'-'*40}")
An HTML header splitter demonstrates how hierarchical headings can guide document partitioning while retaining section context. section 6.4.3
