liveBook · Manning

Requests

As explained in Learn AI Data Engineering

Requests is a Python library for sending HTTP requests and handling their responses.

requests is a Python library for making HTTP calls. Give it a URL and receive a response that can supply API JSON, a downloaded CSV, or a web page. In data pipelines, it is the fetching layer: requests retrieves external data before pandas, HTML parsing, or later normalization work handles it. section 10.5.1 section 4.1.1

Figure 4.1 Interacting with an API: This diagram shows how a client constructs a full request URL by combining a base endpoint with query parameters that define the data request (e.g., filters, limits, and format). The server receives the request, handles tasks like authentication and data lookup, and returns a structured JSON response. This client-server pattern is central to data engineering workflows, where APIs often serve as the primary source of external or cloud-based data. Knowing how to shape requests and interpret responses is essential for working with modern data pipelines.

An HTTP request combines an endpoint with query parameters; the server then returns a structured response such as JSON. This client-server exchange is the basic shape behind API ingestion. section 4.1.1

A practical request loop should treat every response as uncertain. Check the HTTP status before consuming data, parse JSON only after a successful response, and record failures alongside the item that caused them. For interactive notebooks, visible progress messages can complement file logging; for longer-running work, timeouts and retry limits prevent a single stalled request from blocking the job. section 4.1.1

Fetch number trivia into a DataFrame

For each number, call the NumbersAPI URL with requests.get(), test for status code 200, read the text field from response.json(), and append a Number/Fact record. On a non-200 response or an exception, append an error record instead and log the outcome. The resulting rows can be loaded into a pandas DataFrame, preserving both successful facts and failures for review. section 4.1.1

For reusable page fetching, separate the network step from parsing and extraction:

  • Create one requests.Session and attach browser-like headers, including a realistic User-Agent. section 10.8
  • Implement fetch_html(url) with session.get(), a timeout, raise_for_status(), and response.text as the return value. section 10.8
  • Pass the returned HTML to BeautifulSoup, then keep title, price, and variant-specific extraction in small helper functions. section 10.8 section 10.5.1
  • Handle unreliable or multi-variant pages as extraction problems after fetching rather than assuming one page contains one trustworthy value. section 10.8

requests and BeautifulSoup form a classic scraping harness: requests downloads raw HTML, while BeautifulSoup turns that markup into a navigable tree for tags, attributes, and text. This division keeps retrieval reliable and easy to embed in a pipeline, while schema normalization can remain a separate concern. section 10.5.1

How Learn AI Data Engineering covers it

How deep it goes
Demonstrates status checks, JSON parsing, logging, timeouts, retries, sessions, headers, and HTTP error surfacing without covering library internals.
Examples
Uses NumbersAPI trivia records and a product-page scraping lab with CSV input, browser-like headers, and variant-price handling.
What it assumes
Assumes a runnable Python notebook or IDE environment and introduces requests through concrete fetching tasks.
What you can do after
Apply it via requests and requests.Session to retrieve JSON or HTML with failure handling suitable for notebooks and pipelines.
How it connects
Connects requests with pandas, JSON, BeautifulSoup, HTML, HTTP status codes, logging, timeouts, retries, and requests.Session.

Also explained in

Compare how 7 books explain requests →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Learn AI Data Engineering. The section links point to the parts of the book it draws on.