liveBook · Manning

Beautiful Soup

As explained in Learn AI Data Engineering

BeautifulSoup is a Python library for parsing HTML and XML documents into navigable structures and extracting or modifying their contents.

Beautiful Soup is a Python HTML parser from the bs4 package. It turns raw markup into a navigable tree, letting a scraper search tags, inspect attributes, remove unwanted elements, and extract text. It commonly follows an HTTP fetch performed with requests, forming a reliable, pipeline-friendly scraping harness. section 10.5.1 section 10.5.3

Parsing is not the same as normalization. After scripts, styles, navigation, and footers are removed, the remaining text can still mix product details with unrelated page content and vary across sites. Beautiful Soup provides the inspectable structure needed to write extraction logic, while inconsistent schemas and edge cases remain a separate problem. section 10.5.1 section 10.5.3 section 10.6

A practical Beautiful Soup workflow is:

  • Fetch page HTML, then construct a BeautifulSoup object with an HTML parser. section 10.5.1 section 10.5.3
  • Inspect formatted markup or a suspected page region before choosing selectors; work from the actual HTML rather than guessing. section 10.6
  • Remove non-product elements with decompose(), then join stripped_strings when a cleaned text representation is useful. section 10.5.3
  • Extract fields with find() or select_one(), and handle absent elements or alternative markup such as sale versus regular prices. section 10.6 section 10.8

Extracting a product title and price

For a product page, find the h1 and return its stripped text when present. Locate the product price block with select_one("div.product-block__price"), prefer a sale-price span, and otherwise use a regular-price span. Each lookup may return None, so the extraction must guard against missing elements. This works for the observed structure, but another site can require different selectors. section 10.6 section 10.8

Listing 10.3 Parsing and cleaning HTML with BeautifulSoup
from bs4 import BeautifulSoup  #A
 
soup = BeautifulSoup(html_content, 'html.parser')  #B
 
# Remove script and style elements
for element in soup(['script', 'style', 'nav', 'footer']):  #C
    element.decompose()
 
# Extract all text
clean_text = ' '.join(soup.stripped_strings)  #D
 
print(f"Cleaned text: {len(clean_text)} characters")
print(f"\nFirst 500 characters:\n{clean_text[:500]}")

This listing demonstrates the core cleanup pass: parse HTML, decompose script/style/navigation/footer elements, and join the remaining stripped strings into normalized text. section 10.5.3

How Learn AI Data Engineering covers it

How deep it goes
Demonstrates tree parsing, element removal, CSS selection, missing-element guards, and price fallbacks; defers scalable normalization.
Examples
Uses cleaned product-page text and GORUCK title and sale-or-regular-price extraction examples.
What it assumes
Assumes Python web fetching and basic HTML inspection while demonstrating Beautiful Soup parsing from supplied HTML.
What you can do after
Apply it via Beautiful Soup to parse fetched HTML and extract guarded title and price values.
How it connects
Connects requests, HTML, CSS selectors, DOM inspection, regular expressions, JSON, and structured extraction.

Also explained in

Compare how 3 books explain Beautiful Soup →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Learn AI Data Engineering. The section links point to the parts of the book it draws on.