liveBook · Manning

Scraping

As explained in Secure APIs

Web scraping is the automated extraction of content or data from websites or web-accessible services.

Scraping automates the extraction of website content or data. In screen scraping, a third-party application captures a user’s bank credentials, logs in as that user, and retrieves account details or performs operations. section 10.1

Unrestricted scraping can create security and business problems:

  • Exposed credentials give third-party applications broad access to private data and account operations. section 10.1
  • Automated scraping can be difficult to distinguish from legitimate activity, enabling attacks against websites and sensitive business flows. section 10.1 section 4.9
  • Mitigation can include checking user agents, limiting headless or command-line access, serving CAPTCHA challenges, and measuring timing across multistep flows. section 4.9
Figure 10.1 In screen scraping, bank customers share their account credentials with a third-party application that uses those credentials to log in to the bank’s website by impersonating the customer.
figure

Visualizes how customers hand credentials to a third-party application that impersonates them on a bank website. section 10.1

How Secure APIs covers it

How deep it goes
Defines screen scraping, explains its consequences, and outlines several automated-request mitigations.
Examples
Uses bank-account screen scraping, content scraping, scalper bots, and timed multistep flows as examples.
What you can do after
Diagnose automated scraping risks and apply request, client, CAPTCHA, and flow-timing mitigations.
How it connects
Connects scraping with APIs, open banking, credentials, automated attacks, headless browsers, and CAPTCHA.

Also explained in

Compare how 9 books explain scraping →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Secure APIs. The section links point to the parts of the book it draws on.