Chapter 13: Reading and writing files
This introduction explains why developers work with text and binary files and identifies Python modules used for particular kinds of binary storage.
After this chapter you will be able to
- Text and binary file roles. Distinguish the common uses of text files from the role of binary files when text files are insufficient.
- Opening and file objects. Explain that open takes a pathname and returns a file object rather than reading the file immediately, and that Python uses the object to track access to the file.
- Successive line reads. Describe what successive calls to readline return, including whether a returned line contains its newline character.
- Reasons to close files. Explain why a file object should be closed after reading or writing is finished, and why leaving files open is more concerning in larger programs.
- Explicit and automatic file closing. Describe how to close a file object with `close()` and how `with` closes it without an explicit `close()` call.
- File opening modes. Distinguish the read, write, and append modes of open, including the default mode and what happens to existing file contents.
- Line reading and memory use. Distinguish readline, readlines, and iteration over a file object by what each returns or processes, and explain why iteration is useful for large files.
- Text newline handling. Explain how text-mode reading handles different line endings and how the newline argument or binary mode changes that behavior.
- Writing strings and lines. Distinguish write from writelines and explain when newline characters must be supplied in their arguments.
- Binary file read behavior. Explain how opening a file with 'rb' and calling read() retrieves bytes, either all remaining data or up to a specified number of bytes.
- Path file reading and writing. Explain how a Path object can read and write text or binary files, including the binary method names and the limitation on appending.
- Standard streams as file objects. Describe the standard input, output, and error streams exposed by sys and the file-style operations this section uses with them.
- Redirecting and restoring standard output. Trace what happens when sys.stdout is set to a file and explain how to restore it, including the notebook-specific precaution.
- Binary record format strings. Describe how a struct format string specifies the types and sequence of values in one binary record, using the section’s format characters as examples.
- Packing and reading records. Trace the section’s write-and-read process using struct.pack, struct.calcsize, file read, and struct.unpack, including how the reading loop ends.
- Pickling and unpickling objects. Explain how pickle converts Python objects to bytes for storage and restores them later, using pickle.dump and pickle.load.
- Selective state persistence. Describe why an application might pickle a dictionary of selected data rather than its entire state, and how it restores that data.
- Cache persistence across sessions. Explain why pickling a calculation cache can save work across program runs, and describe the example’s initialization and concurrent-save limitations.
- Pickle efficiency and security drawbacks. Explain why pickle may be a poor serialization choice: it is not especially fast or space efficient, and loading a malicious pickle can execute arbitrary code. State when the section says to avoid it.
- Persistent dictionary-like storage. Explain how a shelf acts like a dictionary whose entries persist in a file, using the address-book example to describe adding an entry and finding it in a later session.
- On-demand access to large files. Explain why a shelf can be useful for a large dataset that a program accesses in pieces.
- File or redirected input. Describe how to refactor the wc utility so it processes a file named on the command line when one is supplied, and reads from stdin when no filename is supplied, including when input is redirected.
- Final wc refactoring request. Describe what the AI bot is asked to change in the earlier wc program: start from the chapter 11 version and add input from standard input and output to standard output.
What it builds on
- File objects (chapter 3) · The earlier overview introduces file objects.
- Context handling using the with keyword (chapter 3) · The earlier overview introduces context handling with with.
- Absolute and relative paths (chapter 12) · That section covers absolute and relative paths.
- Manipulating pathnames (chapter 12) · That section covers manipulating pathnames.
- Bytes (chapter 6) · The bytes data type.
- Manipulating pathnames with pathlib (chapter 12) · Using Path objects to work with pathnames.
- Getting input from the user (chapter 4) · Basic input from the user
- Dictionaries as caches (chapter 7) · Caching calculation results in a dictionary keyed by their arguments.
Section by section
Introduction
This introduction explains why developers work with text and binary files and identifies Python modules used for particular kinds of binary storage.
- Text and binary file roles: Distinguish the common uses of text files from the role of binary files when text files are insufficient.
Why it matters: The distinction helps you choose a suitable kind of file for what you need to store. - Binary storage module roles: State the limited roles assigned here to struct, pickle, and shelve in storing binary data.
Why it matters: These names point to different storage needs without requiring you to know their mechanics yet.
Key terms: text file, binary file, struct module, pickle, shelve
13.1 Opening files and file objects
This section explains how opening a file creates a file object, how successive reads get lines from it, and how to supply paths and use the preferred opening style.
- Opening and file objects: Explain that open takes a pathname and returns a file object rather than reading the file immediately, and that Python uses the object to track access to the file.
Why it matters: It helps you distinguish getting access to a file from actually retrieving its contents. - Successive line reads: Describe what successive calls to readline return, including whether a returned line contains its newline character.
Why it matters: Successive reads give you successive lines, and the newline can be part of the text returned. - Preferred with opening style: State why the example uses with when opening a file, without needing to explain context-manager mechanics.
Why it matters: It identifies the opening style the book recommends before explaining context managers in detail. - Pathnames for opening files: Describe how the pathname passed to open identifies a file, and how os.path.join can construct a path using the current operating system's conventions.
Why it matters: The pathname tells open which file to access, while os.path.join helps build paths across operating systems.
Key terms: open, pathname, file object, file I/O, readline, newline character, with, context manager, I/O errors, current working directory
13.2 Closing files
This section explains why file objects should be closed and how to close them explicitly or automatically with `with`.
- Reasons to close files: Explain why a file object should be closed after reading or writing is finished, and why leaving files open is more concerning in larger programs.
Why it matters: Closing files helps keep programs reliable, especially as they grow. - Explicit and automatic file closing: Describe how to close a file object with `close()` and how `with` closes it without an explicit `close()` call.
Why it matters: These are the two ways the section gives to ensure a file is closed when you're done with it.
Key terms: file object, close, system resources, close method, with, context manager
13.3 Opening files in write or other modes
This section explains how the mode argument to open controls reading, writing, and appending, and briefly introduces optional file-handling parameters.
- File opening modes: Distinguish the read, write, and append modes of open, including the default mode and what happens to existing file contents.
Why it matters: Choosing the wrong mode can erase data or put new data in the wrong place. - Optional file opening parameters: Explain what buffering means for file reads and writes, and state what other optional open parameters control.
Why it matters: These options explain why open can accept arguments beyond the basic file mode.
Key terms: open, 'r', 'w', 'a', buffering, encoding, newline characters
13.4 Functions to read and write text or binary data
This section compares ways to read and write file contents, including their memory use and treatment of newline characters.
- Line reading and memory use: Distinguish readline, readlines, and iteration over a file object by what each returns or processes, and explain why iteration is useful for large files.
Why it matters: Choosing how to read lines affects both the shape of your code and how much memory it uses. - Text newline handling: Explain how text-mode reading handles different line endings and how the newline argument or binary mode changes that behavior.
Why it matters: Newline handling can change the line-ending characters you read or write. - Writing strings and lines: Distinguish write from writelines and explain when newline characters must be supplied in their arguments.
Why it matters: Knowing that these methods do not add line breaks prevents accidentally concatenated output.
Key terms: readline, readlines, file object, iterator, text mode, universal mode, newline parameter, binary mode, write, writelines
13.4.1 Using binary mode
Explains how to read file data as bytes in binary mode and when those bytes must be decoded into strings.
- Binary file read behavior: Explain how opening a file with 'rb' and calling read() retrieves bytes, either all remaining data or up to a specified number of bytes.
Why it matters: This lets you choose between reading the remaining file at once and reading a specific number of bytes. - Bytes and string distinction: Distinguish the bytes returned by a binary-mode file from strings, and explain what must happen before using those bytes as strings.
Why it matters: Text-like data may still arrive as bytes, so treating it as a string requires an explicit conversion.
Key terms: binary mode, 'rb', read method, bytes object, bytes objects, string objects, decode
13.5 Reading and writing with pathlib
This section shows how Path objects read and write text or binary files without explicit open and close calls, while noting that writing replaces existing content.
- Path file reading and writing: Explain how a Path object can read and write text or binary files, including the binary method names and the limitation on appending.
Why it matters: Path offers a concise way to handle file contents, but its write methods are unsuitable when you need to append.
Key terms: Path object, read_bytes, write_bytes, text, binary mode, append
13.6 Terminal input/output and redirection
This section explains how to read terminal input and direct printed or stream output to files, including how to restore standard output afterward.
- Terminal input as strings: Explain how input prompts a user, reads a line, and handles values that need to be numbers.
Why it matters: It helps you handle terminal responses without mistaking text for a number. - Standard streams as file objects: Describe the standard input, output, and error streams exposed by sys and the file-style operations this section uses with them.
Why it matters: These streams connect familiar file operations to a program’s terminal input and output. - Redirecting and restoring standard output: Trace what happens when sys.stdout is set to a file and explain how to restore it, including the notebook-specific precaution.
Why it matters: Restoring the stream lets later output return to its intended destination. - Print output to a chosen file: Distinguish redirecting one print call with its file parameter from changing standard output, and describe the section’s capture-and-restore example.
Why it matters: You can choose between sending a single print call to a file and capturing ongoing printed output.
Key terms: input, prompt, int, sys.stdin, sys.stdout, sys.stderr, standard input, standard output, standard error, redirect
13.7 Handling structured binary data with the struct module
The section shows when the struct module is useful and how a format string lets Python write and read structured binary records.
- Structured binary file use cases: Explain why struct is useful when reading or writing binary files generated or used by other programs, rather than for most files you create yourself.
Why it matters: It helps you choose struct for a defined binary format rather than treating it as the default way to save Python data. - Binary record format strings: Describe how a struct format string specifies the types and sequence of values in one binary record, using the section’s format characters as examples.
Why it matters: The same format description is what lets struct interpret the bytes of a record consistently. - Packing and reading records: Trace the section’s write-and-read process using struct.pack, struct.calcsize, file read, and struct.unpack, including how the reading loop ends.
Why it matters: These operations connect a record’s format to both its stored bytes and the Python values recovered from them.
Key terms: struct module, binary data, pickling, format string, C short integer, C double-precision floating-point number, 'hd7s', struct.pack, struct.calcsize, read
13.8 Pickling objects to files
This section explains how pickle saves and restores Python objects, how to choose what to persist, and how a persistent cache can reuse results across program runs.
- Pickling and unpickling objects: Explain how pickle converts Python objects to bytes for storage and restores them later, using pickle.dump and pickle.load.
Why it matters: It lets a program preserve objects without writing separate code to serialize and rebuild each one. - Pickleable objects and limits: Distinguish the kinds of objects pickle can handle from the code objects and system resources it cannot, including how it preserves shared references.
Why it matters: Knowing these limits helps you decide which parts of a program’s state pickle can preserve. - Selective state persistence: Describe why an application might pickle a dictionary of selected data rather than its entire state, and how it restores that data.
Why it matters: Selecting what to save keeps unrelated application state out of a file and makes restoration less error-prone. - Cache persistence across sessions: Explain why pickling a calculation cache can save work across program runs, and describe the example’s initialization and concurrent-save limitations.
Why it matters: A persistent cache can reuse expensive results on later runs, provided the example’s file and concurrency limitations are accounted for.
Key terms: pickle module, pickling, unpickling, byte stream, pickle.dump, pickle.load, shared objects, cyclic references, code objects, system resources
13.8.1 Reasons not to pickle
This section explains why pickle can be a poor choice for storing objects, especially when a file might be altered by someone else.
- Pickle efficiency and security drawbacks: Explain why pickle may be a poor serialization choice: it is not especially fast or space efficient, and loading a malicious pickle can execute arbitrary code. State when the section says to avoid it.
Why it matters: These trade-offs help you decide whether saving an object as a pickle is appropriate.
Key terms: pickling, serialization, JSON, arbitrary code
13.9 Shelving objects
This section explains how shelves provide persistent, dictionary-like access to objects on disk, why they help with large datasets, and where their limits matter.
- Persistent dictionary-like storage: Explain how a shelf acts like a dictionary whose entries persist in a file, using the address-book example to describe adding an entry and finding it in a later session.
Why it matters: A shelf gives you familiar key-based access to data that remains available across sessions. - On-demand access to large files: Explain why a shelf can be useful for a large dataset that a program accesses in pieces.
Why it matters: On-demand access can make large files practical to use without loading them all at startup. - Shelf restrictions and trade-offs: Distinguish shelf objects from ordinary dictionaries by describing their key and value restrictions and the practical drawbacks of writing, concurrent use, and closing.
Why it matters: These limits help you decide when a shelf is convenient and when it is a poor fit.
Key terms: shelve.open, shelf object, persistent dictionary, file accesses, lookups, strings as keys, pickled, concurrent access, close
13.10 Final fixes to wc
This exercise asks readers to make the Python wc utility process either a named file or redirected standard input.
- File or redirected input: Describe how to refactor the wc utility so it processes a file named on the command line when one is supplied, and reads from stdin when no filename is supplied, including when input is redirected.
Why it matters: This lets the same utility work with a filename or with input redirected from the command line.
Key terms: wc utility, input redirection, command-line argument, stdin, argparse, nargs
13.10.1 Solving the problem with AI-generated code
This subsection frames the final wc refactoring as an AI-assisted task and explains why a harder byte-versus-character counting feature was left out.
- Final wc refactoring request: Describe what the AI bot is asked to change in the earlier wc program: start from the chapter 11 version and add input from standard input and output to standard output.
Why it matters: This identifies the change the final wc exercise is meant to make. - Byte and character counting distinction: Explain why counting bytes and counting characters was a separate, harder feature that the author removed from this exercise.
Why it matters: It explains both the counting distinction and why it is not part of the requested refactoring.
Key terms: wc utility, standard input, standard output, -c, -m, bytes, characters, binary mode, text mode
13.10.2 Solutions and discussion
13.11 Summary
Where this is used later
- Text encoding: ASCII, Unicode, and others (chapter 21) · Opening files with Python’s open function.
- Context managers using the with keyword (chapter 14) · That section explains context managers using with in more detail.
- Reading text files (chapter 21) · Provides basic ways to read text files before processing their contents.
- Making an object behave like a list (chapter 17) · Assumes familiarity with reading text from files.
- Writing data files (chapter 21) · Provides the basic behavior of writing text to data files.
- Using Python to fetch files from an FTP server (chapter 22) · Opening a file for binary writing
Concepts explained in this chapter
- input/output redirectionInput redirection is a shell mechanism that supplies a command's standard input from a file or another source instead of an interactive terminal.
- standard inputsys.stdin is the Python standard-stream object that provides text input from the terminal or a redirected source.
- readlinereadline is a file-object method that reads and returns one line, including its terminating newline when present, or an empty string at end of file.
- write operationwrite is a file-object method that writes text or bytes to an open file and returns the number of units written.
- openopen() is Python's built-in function for opening a file and returning a file object.
- readlinesreadlines is a file-object method that reads all remaining lines from an open stream and returns them as a list.
- writelineswritelines is a file-object method that writes the strings from an iterable consecutively without automatically inserting newline characters.
- binary modeBinary mode is a file-access mode in which file operations read or write raw bytes without text encoding or newline translation.
- argparseargparse is Python's standard-library module for parsing command-line arguments and generating related help and error messages.
- shelveshelve is a Python standard-library module that provides persistent dictionary-like storage for Python objects, typically backed by serialization.
- binary dataBinary data is information represented as encoded bytes rather than directly as human-readable text characters.
- picklepickle is a Python standard-library serialization facility for converting Python objects to and from a byte stream.
- struct moduleThe struct module is a Python standard-library module for converting between Python values and packed binary representations.
- I/OI/O is the transfer of data into and out of a program through streams, files, devices, or other interfaces.
- byte streamA byte stream is an ordered sequence of bytes representing serialized data or other binary content.
- cachingA cache is stored data containing previously computed results so that later requests can reuse them without repeating the computation.
- unpicklingUnpickling is the deserialization of a byte stream into a Python object hierarchy.
- shelf objectA shelf object is a file-backed mapping that stores picklable Python objects under string keys and retrieves them through dictionary-like operations.
- shelve.openshelve.open is the shelve module function that opens or creates a file-backed shelf and returns a shelf object.
- calcsizecalcsize is a struct-module function that returns the number of bytes required by a specified format string.
- format stringA format string is a string containing replacement fields that specify how supplied values are incorporated into formatted text.
- recordA row is a horizontally aligned record of related values in a tabular data structure.
- struct.packstruct.pack is a struct-module function that converts supplied Python values into a packed byte sequence according to a format string.
- struct.unpackstruct.unpack is a struct-module function that parses a packed byte sequence according to a format string and returns the resulting Python values as a tuple.
- input()input() is Python's built-in function for obtaining a line of text from the user through standard input.
- standard streamsStandard streams are the conventional input, output, and diagnostic communication channels provided to a running program.
- iteratorAn iterator is an object that provides successive values through the iteration protocol until its values are exhausted.
- newlineA newline is a character or character sequence that marks the boundary between consecutive lines of text.
- file objectA data file is a stored file whose contents provide data for retrieval, processing, or analysis.
- file pathA file path is a textual or structured identifier specifying the location of a file or directory in a filesystem.
- with statementA with statement is a Python control-flow construct that establishes a context manager-controlled block and performs its exit actions when the block ends.
- os.path.joinos.path.join is a Python standard-library function that combines pathname components using the conventions of the current operating system.
- code generationCode generation is the automated production of source code from prompts, specifications, templates, or other inputs.
- wc utilityThe wc utility is a Unix command-line tool that counts lines, words, and bytes in files or input streams and can report the maximum line length.
- serializationSerialization is the transformation of an in-memory data structure or object into a representation suitable for storage or transmission.
- pathlibpathlib is a Python standard-library module that provides object-oriented classes and operations for representing and manipulating filesystem paths.
- command-line argumentA command-line argument is a value supplied to a program when it is invoked through a command-line interface.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.