Scraping
Scraping is the automated or systematic retrieval and extraction of information from digital sources.
Effective Data Analysis
Web scraping is the automated retrieval and extraction of information from web pages, typically by requesting and interpreting their HTML content.
- How it is used
- Augments organizational analyses by converting website information unavailable through an API into reusable datasets.
- How deep it goes
- Explains requests, HTML, tags, BeautifulSoup, extraction, and limitations, while stopping short of advanced scraping architectures.
- Examples
- Uses weather forecasts, competitor prices, social-media trends, financial websites, and book metadata as scraping examples.
- What you can do after
- Apply web scraping with requests and BeautifulSoup to extract HTML fields and assemble structured analytical data.
Practical Automation with PowerShell: Effective scripting from the console to the cloud
Web scraping is the automated extraction of data from web pages by retrieving and parsing their content.
- How it is used
- Automates Microsoft Update Catalog lookups when no public search backend is available.
- How deep it goes
- Builds a complete XPath-driven PowerShell workflow, then tests its external-data assumptions against changing pages.
- Examples
- Uses KB4521858 search results and Microsoft Update Catalog detail pages to extract update metadata and supersedence.
- What you can do after
- Build it from scratch as a reusable PowerShell function that collects and validates update-supersedence data.
Machine Learning Platform Engineering
Metrics scraping is the periodic retrieval of metric data from an application's exposed endpoint.
- How it is used
- Supports production observability by collecting application metrics for health checks, latency monitoring, traffic analysis, dashboards, and alerts.
- How deep it goes
- Explains endpoints, instrumentation, discovery, verification, queries, dashboards, and alerts, but stops short of detailed Prometheus configuration.
- Examples
- Uses a FastAPI /metrics endpoint, custom counters, Kubernetes scrape annotations, Prometheus queries, and Grafana request charts.
- What you can do after
- Operate/configure a pull-based metrics pipeline and validate scraped values before visualizing or alerting on them.
Read how Machine Learning Platform Engineering explains scraping →
Introduction to Generative AI, Second Edition
Web scraping is the automated retrieval and extraction of content from web pages or websites.
- How it is used
- Uses programmatic web access to gather material for datasets that train and test artificial intelligence models.
- How deep it goes
- Defines scraping and sketches its legal, commercial, licensing, and robots.txt dimensions without implementation detail.
- Examples
- Uses ImageNet, search-engine bots, Reddit, Twitter, Stack Overflow, and an outdated Reddit recommendation as examples.
- What you can do after
- Evaluate scraped web data as a dataset source while considering public availability, licenses, bot instructions, and source freshness.
Read how Introduction to Generative AI, Second Edition explains scraping →
WordPress Power Toolkit: Harness AI to build next-level websites
Data scraping is the automated collection of information from websites or other digital sources for storage or analysis.
- How it is used
- Explains how AI systems collect, store, and retrieve information for answers.
- How deep it goes
- Defines automated collection and sketches storage and retrieval, but does not detail scraping implementation.
- Examples
- Uses internet pages, YouTube videos, browser searches, and a recent-events question as examples.
- What you can do after
- Evaluate how AI systems may obtain information from stored data or current browser-based searches.
Read how WordPress Power Toolkit: Harness AI to build next-level websites explains scraping →
AI Agents and Applications
Page scraping is the programmatic extraction of content from web pages.
- How it is used
- Automates source collection for a research summarization engine that searches, extracts, summarizes, and combines web content.
- How deep it goes
- Explains a runnable implementation with metadata flow, character limiting, parallel mapping, and summary joining, but not scraping internals.
- Examples
- Uses Astorga attraction searches, travel URLs, extracted page text, and a source-preserving summary as concrete artifacts.
- What you can do after
- Build it from scratch as a Python and LangChain chain that extracts, limits, summarizes, and combines text from multiple URLs.
The Quick Python Book, Fourth Edition
Web scraping is the automated extraction of data from content delivered by websites or other web resources.
- How it is used
- Uses website content as a fallback when APIs or other data sources do not provide the needed information.
- How deep it goes
- Demonstrates fetching, HTML parsing, selectors, indexing, text, and attributes, but defers complex scraping cases.
- Examples
- Uses a local HTML page containing a special span and link, plus a forecast page with repeated labeled rows.
- What you can do after
- Build it from scratch by parsing HTML with Beautiful Soup and extracting selected elements, text, and attributes.
Read how The Quick Python Book, Fourth Edition explains scraping →
Learn Docker in a Month of Lunches, Second Edition
Scraping is the periodic retrieval of monitoring data from an exposed endpoint.
- How it is used
- Collects application metrics for Prometheus-based observability in containerized deployments.
- How deep it goes
- Explains pull behavior, intervals, endpoints, target discovery, and scaling caveats without detailing Prometheus internals.
- Examples
- Uses image-gallery, iotd-api, and access-log jobs with static targets, DNS discovery, and a ten-second interval.
- What you can do after
- Operate/configure Prometheus scraping for single containers or scaled services using endpoint paths and target discovery.
Read how Learn Docker in a Month of Lunches, Second Edition explains scraping →
Secure APIs
Web scraping is the automated extraction of content or data from websites or web-accessible services.
- How it is used
- Frames scraping as an automated-access risk in open banking and sensitive business flows.
- How deep it goes
- Defines screen scraping, explains its consequences, and outlines several automated-request mitigations.
- Examples
- Uses bank-account screen scraping, content scraping, scalper bots, and timed multistep flows as examples.
- What you can do after
- Diagnose automated scraping risks and apply request, client, CAPTCHA, and flow-timing mitigations.
Related concepts
- Prometheusused by · Prometheus is an open-source monitoring and alerting system that collects, stores, and queries time-series metrics.
- Beautiful Soupuses · Beautiful Soup is a Python library for parsing HTML and XML documents, navigating their parsed structures, and extracting or modifying their contents.
- HTMLuses · HTML (HyperText Markup Language) is the standard markup language for structuring content and elements in web documents.
- contractalternative to · An API contract is a specification of the operations, data, and behavioral expectations that govern interactions between software components.
- requestsuses · Requests is a Python library that provides a high-level interface for sending HTTP requests and handling responses.
- yfinanceused by · yfinance is a Python library for retrieving financial market data from Yahoo Finance through a programmatic interface.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.