liveBook · Manning

HTML cleaning

As explained in Learn AI Data Engineering

HTML cleaning is the removal of irrelevant, distracting, or unsafe markup and content from an HTML document before further processing.

HTML cleaning removes markup and page regions that distract downstream product extraction. It targets rendering-oriented noise, irrelevant content, and empty elements before sending text to an LLM. section 11.5

A practical cleaning pipeline combines three rules:

  • Decompose structural tags such as script, style, nav, header, footer, iframe, noscript, svg, and form. section 11.5
  • Remove containers whose class names indicate breadcrumbs, related products, reviews, cookie banners, carts, headers, or other non-product regions. section 11.5
  • Delete elements without text or images, then join the remaining stripped strings into clean text. section 11.5

Measure the reduction

Run the cleaner on raw_html, compare raw and cleaned character counts, and calculate reduction_pct = (1 - len(clean_text) / len(raw_html)) * 100. Then estimate tokens and cost before a larger extraction batch. section 11.5 section 11.8

Listing 11.3 Tag-level HTML cleaning (improved BeautifulSoup approach)
from bs4 import BeautifulSoup #A
 
def clean_html_aggressive(html: str) -> str: #B
"""Remove common non-product elements and return clean text."""
soup = BeautifulSoup(html, "html.parser")
 
for tag_name in REMOVE_TAGS: #C
for element in soup.find_all(tag_name):
element.decompose()
 
for class_pattern in REMOVE_CLASSES: #D
for element in soup.find_all(
class_=lambda c: c and class_pattern in " ".join(c).lower()
):
element.decompose()
 
for element in soup.find_all(): #E
if not element.get_text(strip=True) and not element.find("img"):
element.decompose()
 
return " ".join(soup.stripped_strings) #F
 
clean_text = clean_html_aggressive(raw_html) #G
 
reduction_pct = (1 - len(clean_text) / len(raw_html)) * 100 #H
print(f"Raw HTML: {len(raw_html):,} characters")
print(f"Cleaned text: {len(clean_text):,} characters")
print(f"Reduction: {reduction_pct:.1f}%")
print(f"Preview: {clean_text[:200]}...") #I

Use the complete BeautifulSoup pattern for tag removal, class-pattern filtering, empty-element cleanup, and normalized text output. section 11.5

How Learn AI Data Engineering covers it

How deep it goes
Defines tag and class rules, implements BeautifulSoup deletion, removes empty elements, and measures character reduction.
Examples
Uses product pages containing navigation, banners, reviews, size guides, shipping information, and related-product modules.
What it assumes
Assumes basic HTML and Python familiarity while teaching reusable tag and class-based cleaning rules from implementation.
What you can do after
Apply it via BeautifulSoup to produce cleaner extraction inputs and estimate savings before processing a larger batch.
How it connects
Connects HTML parsing with BeautifulSoup, product extraction, LLM APIs, token costs, and data pipelines.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Learn AI Data Engineering. The section links point to the parts of the book it draws on.