A document transformer reshapes loaded documents for efficient retrieval. Its practical role is to split long documents into smaller, representative chunks that indexing and language-model systems can process more effectively. section 6.4.3
Splitting a long document
A lengthy HTML document can be divided according to its headings and sections, producing smaller chunks that support more granular indexing and retrieval than the original document. section 6.4.3
Listing 6.2 HTML splitter
# Install langchain-text-splitters if not already installed
%pip install -qU langchain-text-splitters
# Import necessary modules
import langchain
import langchain_text_splitters
print("Langchain version:", langchain.__version__)
print("Langchain Text Splitters module loaded successfully!")
from langchain_text_splitters import HTMLHeaderTextSplitter
from langchain.schema import Document # Ensure Document is properly imported
from bs4 import BeautifulSoup
print("BeautifulSoup is installed successfully!")
# Sample HTML content to be split
html_string = """
<!DOCTYPE html>
<html>
<body>
<div>
<h1>Introduction</h1>
<p>This is the introduction section of the document.</p>
<div>
<h2>Chapter 1: Getting started</h2>
<p>This section covers the basics of getting started.</p>
<h3>Section 1.1 Setup</h3>
<p>This subsection explains the setup process.</p>
<h3>Section 1.2 Configuration</h3>
<p>This subsection details the configuration options.</p>
</div>
<div>
<h2>Chapter 2: Advanced Techniques</h2>
<p>This section dives into more advanced techniques.</p>
</div>
<br/> <!-- Fix: Ensuring self-closing tag is correctly formatted -->
<p>What you learned in the Introduction.</p>
</div>
</body></html>
"""
# Define header tags to split on (h1, h2, h3 represent different levels of headers)
headers_to_split_on = [
("h1", "Header 1"), # Top-level headers
("h2", "Header 2"), # Subsection headers
("h3", "Header 3"), # Sub-subsection headers
]
# Initialize the HTML header text splitter with the specified header levels
html_splitter = HTMLHeaderTextSplitter(headers_to_split_on=headers_to_split_on)
# Split the HTML document into structured chunks
html_header_splits = html_splitter.split_text(html_string)
# Display structured output
for doc in html_header_splits:
print(f"Content:\n{doc.page_content}\nMetadata: {doc.metadata}\n{'-'*40}")
Inspect an HTML splitter example that demonstrates transforming structured HTML into smaller retrieval-oriented pieces. section 6.4.3
