Beautiful Soup parses retrieved HTML into a navigable parser object, avoiding brittle string searches and regular expressions when extracting web data. Fetch the page separately, commonly with requests, then pass its text to BeautifulSoup with an HTML parser. section 22.4
Use the parsed tree to extract tags, attributes, and selected elements:
- Call the parser with a tag name, such as bs("a"), to obtain matching link elements. section 22.4
- Index the result like a list, then read a tag's visible content through .text. section 22.4
- Read attributes with item["href"] or item["class"], receiving the attribute value or class list. section 22.4
- Use select(".special") to find elements matching a class selector; returned elements remain parser objects. section 22.4
Extract a link and a special element
Parse test.html with bs4.BeautifulSoup(html, "html.parser"), then evaluate a_list = bs("a"). The first item exposes a_item.text as 'link' and a_item["href"] as 'http://bitbucket.dev.null'. Evaluating bs.select(".special") finds the span whose text is 'this is special'. section 22.4
Listing 22.1 File test.html
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML//EN"> <html> <head> <title>Title</title> </head> <body> <h1>Heading 1</h1> This is plan text, and is boring <span class="special">this is special</span> Here is a <a href="http://bitbucket.dev.null">link</a> <hr> <address>Ann Address, Somewhere, AState 00000 </address> </body> </html>
Inspect the small HTML fixture containing a heading, ordinary text, a special span, a link, and an address element. section 22.4
