Chapter 20: Basic file wrangling
The introduction frames file wrangling as the planned, automated management of growing collections of files that cannot be discarded immediately.
After this chapter you will be able to
- Planned file lifecycle management. Explain why a growing collection of files needs a plan for saving, managing, and eventually disposing of them without manual intervention.
- Accumulating data file problem. Explain why routinely generated data files become difficult to manage: their data may be worth keeping, so their numbers and storage use grow over time.
- Reasons for archiving daily feeds. Explain why repeatedly arriving product files cannot always replace the previous day’s files and how dating and archiving them addresses that need.
- Pathlib archive workflow. Trace how the example finds incoming text files, constructs date-stamped filenames, and moves them into an archive using pathlib.
- Date-based archive organization. Explain why a separate date-named subdirectory for each received set makes an accumulating archive easier to manage than keeping all files in one directory with dates in their names.
- Subdirectory archiving change. Describe the change to the archiving process: use the date string to create a subdirectory in the archive directory, then move each file into it without adding the date to its filename.
- Limits of adding storage. Explain why acquiring a bigger disk may be an easy, economical response to accumulating data files, especially in the cloud, but does not solve the underlying storage problem.
- Date-named zip archive workflow. Trace how the script archives a day’s files in a zip file instead of moving them into an archive subdirectory.
- Age-based file retention. Explain why archived files may need grooming even after compression and describe the example retention rule.
- Date-based file selection. Trace how the example uses date-derived archive names and Python libraries to identify and remove files that fail its retention rule.
What it builds on
- Manipulating pathnames with pathlib (chapter 12) · Uses pathlib path operations for manipulating filenames and locations.
Section by section
Introduction
The introduction frames file wrangling as the planned, automated management of growing collections of files that cannot be discarded immediately.
- Planned file lifecycle management: Explain why a growing collection of files needs a plan for saving, managing, and eventually disposing of them without manual intervention.
Why it matters: A plan helps you handle accumulating files without having to manage each one by hand.
Key terms: log files, regular data feed, manual intervention
20.1 The problem: The never-ending flow of data files
This section explains why a continuing flow of valuable data files becomes a file-management and storage problem.
- Accumulating data file problem: Explain why routinely generated data files become difficult to manage: their data may be worth keeping, so their numbers and storage use grow over time.
Why it matters: It explains why even ordinary data files eventually need more than manual handling.
Key terms: data files, flat text files, raw data, storage
20.2 Scenario: The product feed from hell
The section uses a daily product-feed example to show why files need archiving and how pathlib can date, rename, and move them.
- Reasons for archiving daily feeds: Explain why repeatedly arriving product files cannot always replace the previous day’s files and how dating and archiving them addresses that need.
Why it matters: Archiving lets a repeatable daily process preserve the original data for later reference. - Pathlib archive workflow: Trace how the example finds incoming text files, constructs date-stamped filenames, and moves them into an archive using pathlib.
Why it matters: This workflow handles recurring files without relying on a fixed list of filenames or manually splitting their names.
Key terms: product feed, raw data, archive folder, pathlib, glob, Path objects, stem, suffix, rename
20.3 More organization
This section replaces a flat, date-renamed file archive with date-named subdirectories that keep each received set together.
- Date-based archive organization: Explain why a separate date-named subdirectory for each received set makes an accumulating archive easier to manage than keeping all files in one directory with dates in their names.
Why it matters: The directory layout determines how easily you can find and manage a received set of files. - Subdirectory archiving change: Describe the change to the archiving process: use the date string to create a subdirectory in the archive directory, then move each file into it without adding the date to its filename.
Why it matters: This is the practical change needed to archive related files as a set.
Key terms: archive, subdirectory, date received, set of files, date string, archive directory, filename
20.4 Saving storage space: Compression and grooming
The section explains why accumulating data files create a storage problem that adding disk space only postpones.
- Limits of adding storage: Explain why acquiring a bigger disk may be an easy, economical response to accumulating data files, especially in the cloud, but does not solve the underlying storage problem.
Why it matters: It distinguishes a convenient short-term response from a lasting solution to growing storage needs.
Key terms: data files, storage, cloud-based platform
20.4.1 Compressing files
This section replaces daily archive subdirectories with date-named zip files to save storage space.
- Compression for daily archives: Explain when compressing archived data files is worth considering and what storage benefit the section expects.
Why it matters: It explains why the archiving script changes from storing files in directories to storing them in zip files. - Date-named zip archive workflow: Trace how the script archives a day’s files in a zip file instead of moving them into an archive subdirectory.
Why it matters: Writing a file into an archive leaves the original in place, so the script must remove it separately.
Key terms: compression, zip file, date string, zipfile library, zipfile object, unlink
20.4.2 Grooming files
This section explains when to remove older archived data files and how to use Python to retain selected files by date.
- Age-based file retention: Explain why archived files may need grooming even after compression and describe the example retention rule.
Why it matters: A retention rule can control storage use when keeping every archived dataset is unnecessary. - Grooming script timing: Describe how the desired grooming interval affects whether grooming belongs with the archiving script or in a separate script.
Why it matters: Choosing when to run grooming helps determine how to organize the scripts. - Date-based file selection: Trace how the example uses date-derived archive names and Python libraries to identify and remove files that fail its retention rule.
Why it matters: Dates in filenames give the script enough information to apply the retention rule automatically.
Key terms: grooming, archived copy, grooming script, archiving script, datetime, pathlib, glob, stem, strptime, timedelta
20.5 Summary
Concepts explained in this chapter
- file archivingFile archiving is the practice of collecting and preserving files in a managed archive for storage or later retrieval.
- directory structureA directory is a filesystem container that organizes files and other directories under a named location.
- file extensionA file extension is a suffix in a filename, commonly separated by a dot, that conventionally indicates a file type.
- file renamingrename is a filesystem path operation that changes a file or directory name and can relocate it when the destination path differs.
- pathlibpathlib is a Python standard-library module that provides object-oriented classes and operations for representing and manipulating filesystem paths.
- file pathA file path is a textual or structured identifier specifying the location of a file or directory in a filesystem.
- glob moduleThe glob module is a Python standard-library module that finds filesystem pathnames matching shell-style wildcard patterns.
- file objectA data file is a stored file whose contents provide data for retrieval, processing, or analysis.
- databaseA database is an organized collection of persistent data managed for storage, retrieval, and modification.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.