Beautiful Soup is a Python library that parses HTML into a navigable tree, making web-page content easier to search, select, clean, and transform into structured data. section 10.2.1
It supports web-scraping workflows by locating repeated tags, sections, links, paragraphs, classes, and attributes without requiring the exact position of each element in the document. section 10.2.1 section 10.2.2
A practical extraction workflow:
- Request the HTML page, initialize BeautifulSoup with an HTML parser, and inspect a prettified version of the document. section 10.2.1
- Find repeated containers or specific tags by name and class, then retrieve their text or selected attributes. section 10.2.2
- Collect extracted values into records, build a dataframe, convert strings to usable data types, and aggregate the results. section 10.2.2
- Check pagination and robots.txt before scraping, and consider how frequently collection should occur when analyzing another organization’s data. section 10.2.4
Weather forecast extraction
A weather page is parsed into forecast containers. Period names, temperatures, and wind descriptions are selected from nested tags, converted into numeric values, and grouped into daily low, high, and wind-speed fields. section 10.2.2
from bs4 import BeautifulSoup #1 soup = BeautifulSoup(response.text, "html.parser") #2 print(soup.prettify()) #3
Inspect the request content by constructing a BeautifulSoup object and printing prettified HTML to reveal its nested structure. section 10.2.1
