Long documents can exceed an LLM’s context window, while oversized passages may include less relevant material. A text splitter divides loaded text into smaller chunks that fit the model and support focused retrieval. section 6.4.3
Choose splitting rules according to the document’s structure:
- Use character- or token-based rules when chunk size is the main constraint. section 6.4.3
- Use HTML, Markdown, or code-aware rules when the document’s structure carries meaning. section 6.4.3
- Prefer smaller, highly relevant chunks over assuming that longer passages improve language-model performance. section 6.4.3
Listing 6.2 HTML splitter
# Install langchain-text-splitters if not already installed
%pip install -qU langchain-text-splitters
# Import necessary modules
import langchain
import langchain_text_splitters
print("Langchain version:", langchain.__version__)
print("Langchain Text Splitters module loaded successfully!")
from langchain_text_splitters import HTMLHeaderTextSplitter
from langchain.schema import Document # Ensure Document is properly imported
from bs4 import BeautifulSoup
print("BeautifulSoup is installed successfully!")
# Sample HTML content to be split
html_string = """
<!DOCTYPE html>
<html>
<body>
<div>
<h1>Introduction</h1>
<p>This is the introduction section of the document.</p>
<div>
<h2>Chapter 1: Getting started</h2>
<p>This section covers the basics of getting started.</p>
<h3>Section 1.1 Setup</h3>
<p>This subsection explains the setup process.</p>
<h3>Section 1.2 Configuration</h3>
<p>This subsection details the configuration options.</p>
</div>
<div>
<h2>Chapter 2: Advanced Techniques</h2>
<p>This section dives into more advanced techniques.</p>
</div>
<br/> <!-- Fix: Ensuring self-closing tag is correctly formatted -->
<p>What you learned in the Introduction.</p>
</div>
</body></html>
"""
# Define header tags to split on (h1, h2, h3 represent different levels of headers)
headers_to_split_on = [
("h1", "Header 1"), # Top-level headers
("h2", "Header 2"), # Subsection headers
("h3", "Header 3"), # Sub-subsection headers
]
# Initialize the HTML header text splitter with the specified header levels
html_splitter = HTMLHeaderTextSplitter(headers_to_split_on=headers_to_split_on)
# Split the HTML document into structured chunks
html_header_splits = html_splitter.split_text(html_string)
# Display structured output
for doc in html_header_splits:
print(f"Content:\n{doc.page_content}\nMetadata: {doc.metadata}\n{'-'*40}")
An HTMLHeaderTextSplitter example demonstrates structure-aware processing for HTML content. section 6.4.3
