Chapter 22: Data over the network
Introduces the chapter’s coverage of moving and obtaining data over a network, including file transfers, APIs, structured formats, and web scraping.
After this chapter you will be able to
- Reasons to automate file retrieval. Explain when fetching data files with Python may be preferable to retrieving them manually.
- Choosing Python over simpler tools. Describe the considerations for choosing a Python script instead of a shell script or command-line tool to retrieve files.
- FTP connection and navigation. Describe how to access an FTP server with ftplib: create an FTP object, connect, log in, and use the connected object to list or change directories.
- Binary file retrieval. Explain what ftp.retrbinary needs to download a remote file and how the example saves the received data locally.
- SFTP and SSH relationship. Explain why SFTP is used when transferred data needs more security, and distinguish it from FTP.
- HTTP file retrieval choices. Explain when Python code is useful for retrieving files over HTTP or HTTPS, and describe how the section uses requests to fetch them.
- Response body forms. Distinguish the response's text and content properties, and describe what you might do with the retrieved data.
- Dynamic API data. Explain how fetching data from an API resembles retrieving a file over HTTP/HTTPS, while the data may be assembled dynamically rather than served from a static file.
- GET requests and query strings. Describe how a common RESTful API uses GET to fetch data and how query-string parameters select the requested data.
- Structured API data formats. Explain that APIs more commonly serve data in structured formats than as plain text, and that JSON and XML are common examples. Describe how both use plain text while structuring it to hold more complex information.
- JSON structures and values. Describe JSON’s two structures and the kinds of keys and values they permit.
- Parsing serialized JSON. Explain why retrieved JSON needs parsing, and distinguish json.loads() from json.load().
- Writing and formatting JSON. Distinguish json.dump() from json.dumps() and describe how to make their output easier to read.
- Storing multiple JSON items. Explain two ways to store multiple items as JSON and how each must be loaded.
- XML structure and extraction challenges. Explain how nested elements, metadata, and data stored in either attributes or text make XML more complex to extract from than a simple delimited file.
- XML to dictionary mapping. Describe how to parse an XML-formatted string with xmltodict and how its result represents attributes, mixed text, and repeated elements.
- Scraping workflow and HTML parsing. Explain the two parts of scraping a website and why the section uses an HTML parser rather than searching the page text by hand.
- Tag and class extraction. Describe how the example uses a Beautiful Soup parser object to find links by tag and elements by class, and what it can do with the returned elements.
- Monthly temperature data workflow. Describe how to request a month of daily mean temperatures for a chosen location using the archive API, then transform the returned dates and temperatures into data suitable for a spreadsheet graph.
- AI suitability for the lab. Explain why the section expects AI to do well on this lab: it has no complex design component and instead calls for accessing an HTTP API and handling the JSON returned.
What it builds on
- Library and third-party modules (chapter 10) · Using modules from Python’s standard library
- Using binary mode (chapter 13) · Opening a file for binary writing
- Installing Python libraries using pip and venv (chapter 19) · Installing a Python library with pip.
- Bytes (chapter 6) · The distinction between bytes and text.
Section by section
Introduction
Introduces the chapter’s coverage of moving and obtaining data over a network, including file transfers, APIs, structured formats, and web scraping.
22.1 Fetching files
This section explains when file retrieval is worth automating and how to decide whether Python is the right tool for it.
- Reasons to automate file retrieval: Explain when fetching data files with Python may be preferable to retrieving them manually.
Why it matters: It helps you decide whether writing retrieval code is worth the effort. - Choosing Python over simpler tools: Describe the considerations for choosing a Python script instead of a shell script or command-line tool to retrieve files.
Why it matters: It keeps a retrieval task from becoming more complicated than it needs to be.
Key terms: data files, automate fetching, Python script, shell scripts, command-line tools, retrieval process
22.1.1 Using Python to fetch files from an FTP server
This subsection shows how to connect to an FTP server with Python, retrieve a file, and optionally use TLS encryption.
- FTP connection and navigation: Describe how to access an FTP server with ftplib: create an FTP object, connect, log in, and use the connected object to list or change directories.
Why it matters: These are the steps that put your program in a position to find and fetch a server’s files. - Binary file retrieval: Explain what ftp.retrbinary needs to download a remote file and how the example saves the received data locally.
Why it matters: The handling method is what turns received FTP data into a local file. - FTP security option: Distinguish the section’s ordinary FTP use case from its stated option for a TLS-encrypted connection.
Why it matters: The choice between FTP and FTP_TLS depends on whether the connection needs encryption.
Key terms: ftplib, FTP object, anonymous, ftp.retrbinary, remote server, binary writing, write method, FTP, Transport Layer Security (TLS), FTP_TLS
22.1.2 Fetching files with SFTP
This subsection explains why SFTP is used for secure file transfers and how Python supports it through paramiko.
- SFTP and SSH relationship: Explain why SFTP is used when transferred data needs more security, and distinguish it from FTP.
Why it matters: The similar names can hide an important difference when choosing how to transfer sensitive data. - Python SFTP tooling: Describe how the section suggests handling SFTP in Python and where SCP fits instead.
Why it matters: It points you to an available Python library without implying that it also provides an SCP function.
Key terms: SFTP, FTP, Secure Shell (SSH), paramiko, pip, SFTP/SCP client module, scp
22.1.3 Retrieving files over HTTP/HTTPS
This section shows how to retrieve a file from a web server with Python and access the returned data.
- HTTP file retrieval choices: Explain when Python code is useful for retrieving files over HTTP or HTTPS, and describe how the section uses requests to fetch them.
Why it matters: You can choose a simple download tool or fetch the file as part of a Python workflow. - Response body forms: Distinguish the response's text and content properties, and describe what you might do with the retrieved data.
Why it matters: Choosing the right form of the response body lets you save or process what the server returned.
Key terms: HTTP/HTTPS, requests, GET, wget, curl, response body, text, content, string, bytes
22.2 Fetching data via an API
Explains how HTTP-based APIs serve data dynamically and how to fetch selected data with GET requests and the requests library.
- Dynamic API data: Explain how fetching data from an API resembles retrieving a file over HTTP/HTTPS, while the data may be assembled dynamically rather than served from a static file.
Why it matters: This distinction explains why an API request can return data selected for that request. - GET requests and query strings: Describe how a common RESTful API uses GET to fetch data and how query-string parameters select the requested data.
Why it matters: The query string lets you request particular records or fields rather than just accept a fixed response. - Requests fetching and URL escaping: Explain how to fetch API data with the requests library and why query parameters containing spaces or punctuation need URL escaping.
Why it matters: Proper URL quoting lets requests send selection parameters that contain characters such as spaces.
Key terms: API, HTTP/HTTPS, static files, RESTful, GET request, query string, query string parameters, requests library, query parameters, quoting
22.3 Structured data formats
Explains why API data commonly uses structured text formats and identifies JSON and XML as the two most common examples.
- Structured API data formats: Explain that APIs more commonly serve data in structured formats than as plain text, and that JSON and XML are common examples. Describe how both use plain text while structuring it to hold more complex information.
Why it matters: This distinction explains why API responses often arrive as JSON or XML rather than unstructured text.
Key terms: API, structured file format, JSON, XML, plain text
22.3.1 JSON data
This section explains how JSON represents data and how to parse, write, and organize JSON data with Python.
- JSON structures and values: Describe JSON’s two structures and the kinds of keys and values they permit.
Why it matters: These rules help you recognize what JSON can represent before converting it to Python data. - Parsing serialized JSON: Explain why retrieved JSON needs parsing, and distinguish json.loads() from json.load().
Why it matters: Parsing turns retrieved or stored JSON into Python data you can work with. - Requests JSON shortcut: Describe the requests response method that replaces an explicit json.loads() call.
Why it matters: The shortcut makes common API-response handling simpler. - Writing and formatting JSON: Distinguish json.dump() from json.dumps() and describe how to make their output easier to read.
Why it matters: Choosing the right function determines whether JSON goes into a file or becomes a string. - Storing multiple JSON items: Explain two ways to store multiple items as JSON and how each must be loaded.
Why it matters: The way you group items determines whether the whole file can be parsed at once.
Key terms: JSON, structures, arrays, serialized, json.loads(), json.load(), requests, response object, json(), json.dump()
22.3.2 XML data
Explains why XML can be difficult to extract data from and how xmltodict turns it into Python dictionaries and lists.
- XML structure and extraction challenges: Explain how nested elements, metadata, and data stored in either attributes or text make XML more complex to extract from than a simple delimited file.
Why it matters: Recognizing where XML stores values helps you avoid missing data during extraction. - XML to dictionary mapping: Describe how to parse an XML-formatted string with xmltodict and how its result represents attributes, mixed text, and repeated elements.
Why it matters: The mapping lets you work with XML data using familiar Python dictionaries and lists.
Key terms: XML, elements, attributes, tree structure, metadata, xmltodict, parse, dictionary, @, #text
22.4 Scraping web data
This section introduces web scraping as fetching a page and extracting data from its HTML, using Beautiful Soup for a simple example while cautioning about legality and site load.
- Scraping workflow and HTML parsing: Explain the two parts of scraping a website and why the section uses an HTML parser rather than searching the page text by hand.
Why it matters: Separating retrieval from extraction helps you choose an appropriate tool for each part. - Tag and class extraction: Describe how the example uses a Beautiful Soup parser object to find links by tag and elements by class, and what it can do with the returned elements.
Why it matters: These two ways of selecting elements let you narrow a page down to the data you want. - Responsible scraping considerations: Explain the two cautions the section gives before scraping a production website you do not control.
Why it matters: A scraper can affect both the site owner’s rights and the site’s availability.
Key terms: scraping, crawling, requests, Beautiful Soup, parser object, HTML tag, select, class, terms of use, denial-of-service attack
22.5 Tracking the weather
This section sets a task to retrieve a month of daily mean temperatures from the Open-Meteo archive API and reshape the results for spreadsheet graphing.
- Monthly temperature data workflow: Describe how to request a month of daily mean temperatures for a chosen location using the archive API, then transform the returned dates and temperatures into data suitable for a spreadsheet graph.
Why it matters: Pairing each date with its temperature makes the API results usable for graphing over time.
Key terms: archive API, latitude, longitude, start_date, end_date, daily=temperature_2m_mean
22.5.1 Solving the problem with AI-generated code
This subsection explains why the weather-tracking lab is a suitable task for AI-generated code.
- AI suitability for the lab: Explain why the section expects AI to do well on this lab: it has no complex design component and instead calls for accessing an HTTP API and handling the JSON returned.
Why it matters: This explains the section’s expectation of AI without claiming that AI can solve every part of a programming task.
Key terms: HTTP API, JSON, AI
22.5.2 Solutions and discussion
22.6 Summary
Concepts explained in this chapter
- requestsRequests is a Python library that provides a high-level interface for making HTTP requests.
- jsonJSON is a Python library for encoding and decoding JSON data.
- xmlXML is a text-based markup and data-interchange format that represents structured information using nested elements and attributes.
- scrapingWeb scraping is the automated extraction of data from content delivered by websites or other web resources.
- json.dump()json.dump() is a Python function that serializes a Python value as JSON and writes it to a file-like object.
- json.dumps()json.dumps() is a Python function that serializes a Python value as a JSON-formatted string.
- json.loads()json.loads() is a Python function that parses a JSON string or bytes object and returns the corresponding Python value.
- serializationSerialization is the transformation of an in-memory data structure or object into a representation suitable for storage or transmission.
- HTTP GETGET is an HTTP method for requesting a representation of a resource without requesting a state-changing operation.
- Beautiful SoupBeautiful Soup is a Python library for parsing HTML and XML documents and navigating their parsed trees.
- HTMLHTML is the HyperText Markup Language, a standard markup language for structuring documents and elements on the web.
- iteratorAn iterator is an object that provides successive values through the iteration protocol until its values are exhausted.
- list()The list function is a Python built-in constructor that creates a new list from an iterable or sequence of elements.
- zipzip is a Python built-in function that aggregates corresponding elements from two or more iterables into tuples, stopping when the shortest iterable is exhausted.
- JSONJSON is a text-based data interchange format that represents structured values using objects, arrays, strings, numbers, booleans, and null.
- PythonPython is a high-level general-purpose programming language with an interpreted execution model and dynamic features.
- file objectA data file is a stored file whose contents provide data for retrieval, processing, or analysis.
- SQLAn SQL query is a statement written in SQL that requests, filters, combines, or modifies data in a database.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.