Web scraping automates data extraction from website content when the information is unavailable elsewhere. The process has two parts: fetch the page, then extract data from its HTML. section 22.4
A compact Beautiful Soup workflow can:
- Read HTML and parse it with Beautiful Soup. section 22.4
- Find links with the a selector, then read their text and href attributes. section 22.4
- Select elements by class, then inspect their text and class attributes. section 22.4
Extracting a link and a special element
Given HTML containing <span class="special">this is special</span> and a link, select links with bs("a"), access the first result with [0], and read a_item.text or a_item["href"]. Select the special span with bs.select(".special"), then inspect its text or class. section 22.4
Listing 22.1 File test.html
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML//EN"> <html> <head> <title>Title</title> </head> <body> <h1>Heading 1</h1> This is plan text, and is boring <span class="special">this is special</span> Here is a <a href="http://bitbucket.dev.null">link</a> <hr> <address>Ann Address, Somewhere, AState 00000 </address> </body> </html>
Inspect the small HTML fixture used to demonstrate link and class-based extraction. section 22.4
Scraping requires practical restraint: scraping an uncontrolled site may be legally or ethically uncertain, and repeated requests can overload a smaller server. section 22.4 section 22.6
