Beautiful Soup is a Python HTML parser from the bs4 package. It turns raw markup into a navigable tree, letting a scraper search tags, inspect attributes, remove unwanted elements, and extract text. It commonly follows an HTTP fetch performed with requests, forming a reliable, pipeline-friendly scraping harness. section 10.5.1 section 10.5.3
Parsing is not the same as normalization. After scripts, styles, navigation, and footers are removed, the remaining text can still mix product details with unrelated page content and vary across sites. Beautiful Soup provides the inspectable structure needed to write extraction logic, while inconsistent schemas and edge cases remain a separate problem. section 10.5.1 section 10.5.3 section 10.6
A practical Beautiful Soup workflow is:
- Fetch page HTML, then construct a BeautifulSoup object with an HTML parser. section 10.5.1 section 10.5.3
- Inspect formatted markup or a suspected page region before choosing selectors; work from the actual HTML rather than guessing. section 10.6
- Remove non-product elements with decompose(), then join stripped_strings when a cleaned text representation is useful. section 10.5.3
- Extract fields with find() or select_one(), and handle absent elements or alternative markup such as sale versus regular prices. section 10.6 section 10.8
Extracting a product title and price
For a product page, find the h1 and return its stripped text when present. Locate the product price block with select_one("div.product-block__price"), prefer a sale-price span, and otherwise use a regular-price span. Each lookup may return None, so the extraction must guard against missing elements. This works for the observed structure, but another site can require different selectors. section 10.6 section 10.8
Listing 10.3 Parsing and cleaning HTML with BeautifulSoup
from bs4 import BeautifulSoup #A
soup = BeautifulSoup(html_content, 'html.parser') #B
# Remove script and style elements
for element in soup(['script', 'style', 'nav', 'footer']): #C
element.decompose()
# Extract all text
clean_text = ' '.join(soup.stripped_strings) #D
print(f"Cleaned text: {len(clean_text)} characters")
print(f"\nFirst 500 characters:\n{clean_text[:500]}")
This listing demonstrates the core cleanup pass: parse HTML, decompose script/style/navigation/footer elements, and join the remaining stripped strings into normalized text. section 10.5.3


