HTML cleaning removes markup and page regions that distract downstream product extraction. It targets rendering-oriented noise, irrelevant content, and empty elements before sending text to an LLM. section 11.5
A practical cleaning pipeline combines three rules:
- Decompose structural tags such as script, style, nav, header, footer, iframe, noscript, svg, and form. section 11.5
- Remove containers whose class names indicate breadcrumbs, related products, reviews, cookie banners, carts, headers, or other non-product regions. section 11.5
- Delete elements without text or images, then join the remaining stripped strings into clean text. section 11.5
Measure the reduction
Run the cleaner on raw_html, compare raw and cleaned character counts, and calculate reduction_pct = (1 - len(clean_text) / len(raw_html)) * 100. Then estimate tokens and cost before a larger extraction batch. section 11.5 section 11.8
Listing 11.3 Tag-level HTML cleaning (improved BeautifulSoup approach)
from bs4 import BeautifulSoup #A
def clean_html_aggressive(html: str) -> str: #B
"""Remove common non-product elements and return clean text."""
soup = BeautifulSoup(html, "html.parser")
for tag_name in REMOVE_TAGS: #C
for element in soup.find_all(tag_name):
element.decompose()
for class_pattern in REMOVE_CLASSES: #D
for element in soup.find_all(
class_=lambda c: c and class_pattern in " ".join(c).lower()
):
element.decompose()
for element in soup.find_all(): #E
if not element.get_text(strip=True) and not element.find("img"):
element.decompose()
return " ".join(soup.stripped_strings) #F
clean_text = clean_html_aggressive(raw_html) #G
reduction_pct = (1 - len(clean_text) / len(raw_html)) * 100 #H
print(f"Raw HTML: {len(raw_html):,} characters")
print(f"Cleaned text: {len(clean_text):,} characters")
print(f"Reduction: {reduction_pct:.1f}%")
print(f"Preview: {clean_text[:200]}...") #I
Use the complete BeautifulSoup pattern for tag removal, class-pattern filtering, empty-element cleanup, and normalized text output. section 11.5
