Chapter 7: Files and storage

Introduction

The introduction explains why permanent data storage is more involved than keeping data in RAM, then previews the chapter’s projects.

• RAM to persistent storage: Explain why saving data for later retrieval involves more than holding it in RAM: information must move from ephemeral electrical charges to storage media and be retrieved through several layers of software indirection.
  Why it matters: This distinction explains why storing data permanently is trickier than it first appears.

Key terms: RAM, storage media, software indirection

7.1 What is a file format?

The section defines file formats as ordered byte sequences and explains the trade-offs among different representations.

• Ordered byte sequence: Explain that a file format is a standard for representing data as one ordered sequence of bytes, and why serial storage favors reading or writing large blocks.
  Why it matters: This explains why storing data involves choices about its byte layout.
• File format design trade-offs: Describe how file formats can trade performance against human-readability and portability, including the contrast between portable, self-describing formats and fast, environment-specific ones.
  Why it matters: No one representation optimizes every useful property of a file format.

Key terms: file formats, ordered sequence of bytes, storage media, performance, human-readability, portability, self-describing

7.2 Creating your own file formats for data storage

The section contrasts recommended long-term database storage with common plain-text files and notes Rust’s support for on-disk formats.

• Long-term storage choices: Distinguish the recommended choice for data stored over a long time from the common use of plain-text files, including why configuration files often use plain text.
  Why it matters: It puts custom file formats in context: common file-based storage is not necessarily the recommended default for long-lived data.
• Rust on-disk format support: State what support the Rust ecosystem offers for storing data in on-disk formats.
  Why it matters: That support makes the following example of writing data in a particular format possible.

Key terms: battle-tested database, plain text files, configuration files, human-readable, machine-readable, Rust ecosystem, on-disk formats

7.2.1 Writing data to disk with serde and the bincode format

This section introduces serde for converting a Rust struct into several storage formats and compares readable and compact encodings.

• Serde struct serialization: Explain that serde can serialize and deserialize Rust values, and that a struct can derive its Serialize and Deserialize traits before being converted to an output format such as JSON.
  Why it matters: It shows how a Rust value can become data suitable for storage without writing a format converter from scratch.
• Encoding format tradeoff: Distinguish JSON from CBOR and bincode in this example: CBOR and bincode are more compact, but unlike JSON they are machine-readable only.
  Why it matters: The format you choose affects whether stored data is easy for people to inspect or compact for machines to handle.

Key terms: serde, Serialize, Deserialize, derive, City struct, JSON, CBOR, bincode, human-readable, machine-readable

7.3 Implementing a hexdump clone

The section builds a small hexdump-style file viewer, moving from string input to a real file while explaining byte reads and command-line arguments.

• Hexadecimal file view: Explain what fview displays and how its hexadecimal output can offer clues about a file’s contents.
  Why it matters: It explains what the viewer’s output represents and why someone might inspect it.
• Mocked and real input: Describe how the first fview example uses a string literal in place of file input and how the next version changes its source to a real file.
  Why it matters: The mock input lets the viewer’s output be demonstrated before file I/O is added.
• Exact-read loop behavior: Trace how the file-reading loop uses read_exact and explain the consequence of requesting more bytes than remain.
  Why it matters: The read size determines both when the loop stops and what happens to an incomplete final read.
• Command-line argument handling: Describe how fview retrieves a command-line argument and what expect does when the requested argument is missing.
  Why it matters: It shows how the example obtains its file argument while making its missing-input behavior and validation limit clear.

Key terms: hexdump, fview, byte, hexadecimal, string literal, &[u8], std::io::prelude, std::io::Read, while let Ok(_), read_exact

7.4 File operations in Rust

The section shifts attention from byte sequences to files and introduces a discussion of more granular file operations.

7.4.1 Opening a file in Rust and controlling its file mode

This subsection explains what the OS provides through files and how Rust opens or creates them with different access modes.

• File Abstraction and Permissions: Explain that the OS presents files as named, hierarchical objects over raw bytes and enforces permissions that restrict access between users.
  Why it matters: Opening a file means working through an OS abstraction that also controls who may access it.
• Basic File Opening Methods: Distinguish the basic roles of std::fs::File, open(), and create(), including when the section says to use open().
  Why it matters: Choosing the basic file operation starts with whether you expect the file to exist.
• Configurable File Access Modes: Describe why std::fs::OpenOptions is used and what access and creation behavior the example requests.
  Why it matters: OpenOptions lets a program request the particular access and preservation behavior its file needs.

Key terms: files, operating system (OS), permissions, std::fs::File, open(), create(), std::fs::OpenOptions, append mode, writeable, readable

7.4.2 Interacting with the filesystem in a type-safe manner with std::fs::Path

This section introduces Rust’s path types and explains why they are preferable to manipulating filesystem paths as strings.

• Path Types and Conversion: Describe how Path and PathBuf relate to str and String, how paths can be created from strings, and what kinds of filesystem locations they can represent.
  Why it matters: It helps you recognize when a value represents a filesystem path rather than ordinary text.
• Reasons for Path Types: Explain why using Path is preferable to manipulating path strings directly, including clarity, portability, and the risk of subtle string-handling bugs.
  Why it matters: Dedicated path operations make code easier to read and less dependent on the conventions of one operating system.

Key terms: Path, PathBuf, from(), set_extension(), path separators, portability

7.5 Implementing a key-value store with a log-structured, append-only storage architecture

Introduces log-structured, append-only databases as a case study in resilient storage and fast reads.

• Append-only database design aims: Explain why the section presents log-structured, append-only databases as significant case studies, including their stated aims despite the unreliability of storage media.
  Why it matters: These stated aims explain why the book uses this database model as a case study.

Key terms: log-structured, append-only model, resilient, optimal read performance

7.5.1 The key-value model

This section defines actionkv’s key-value model and explains the kinds of queries it supports.

• Key value storage model: Explain that actionkv stores and retrieves arbitrary-length byte sequences as keys and values, allowing a value to be looked up by its key but not supporting broader queries across entries.
  Why it matters: This distinction sets expectations for what actionkv can retrieve.

Key terms: key-value model, key, value, actionkv, [u8]

7.5.2 Introducing actionkv v1: An in-memory key-value store with a command-line interface

The section explains why actionkv starts as a library crate and how its Cargo configuration identifies an executable that uses the shared library.

• Shared actionkv library crate: Explain why actionkv starts with `cargo new --lib actionkv` and how its `lib.rs` file will be used by multiple executables.
  Why it matters: This explains why actionkv separates shared library code from its executables.
• Repeatable executable declarations: Describe how actionkv identifies its executable binaries in Cargo.toml and what the double brackets around `bin` mean.
  Why it matters: The repeatable declarations let the project identify its executables without ambiguity.

Key terms: library crate, lib.rs, cargo new --lib actionkv, Cargo.toml, [[bin]], executable binaries

7.6 Actionkv v1: The front-end code

This section identifies actionkv v1’s public operations and explains how its command-line front end dispatches requests while delegating storage work.

• Front-end dispatch and delegation: Explain that the front end uses Rust’s matching facilities to select an internal function from command-line arguments, while an ActionKV instance named store handles the filesystem work.
  Why it matters: This separates what the command-line code does from the storage code explained next.
• Public operations and naming: State actionkv’s four public operations and explain why their names follow std::collections::HashMap rather than choosing among alternatives such as retrieve or fetch.
  Why it matters: The operation names make the front end’s available requests clear without inventing a separate naming convention.

Key terms: matching facilities, command-line arguments, ActionKV, store, get, delete, insert, update, std::collections::HashMap

7.6.1 Tailoring what is compiled with conditional compilation

This section explains how Rust uses cfg attributes to compile target-specific code and how to combine or supply cfg conditions.

• Target-specific conditional compilation: Explain how cfg attributes select code at compile time for a target, and describe the section’s Windows-specific USAGE example.
  Why it matters: It lets one source project produce binaries containing only the code and data appropriate to their targets.
• Cfg condition expressions: Describe the cfg forms for negating or combining matches and how a cfg attribute can be supplied on the compiler command line.
  Why it matters: These forms let you express which builds should include a piece of source code.

Key terms: conditional compilation, cfg attributes, target, const USAGE, not, all, any, --cfg ATTRIBUTE

7.7 Understanding the core of actionkv: The libactionkv crate

This section identifies libactionkv::ActionKV as the component that carries out the command-line application's storage work.

• ActionKV component responsibilities: Explain how the command-line application relates to libactionkv::ActionKV and what the struct is responsible for.
  Why it matters: This division of responsibilities clarifies which component handles the database's storage work.

Key terms: libactionkv::ActionKV, ActionKV struct, on-disk format

7.7.1 Initializing the ActionKV struct

This section explains how ActionKV opens its data file and rebuilds an in-memory index, introduces byte-oriented type aliases, and clarifies how a file read signals EOF.

• ActionKV initialization steps: Describe how creating an ActionKV instance opens the data file and loads offsets for existing data into an in-memory index, and note that both steps return Result.
  Why it matters: This connects the stored data to the index ActionKV uses after opening a database.
• Binary string type aliases: Distinguish ByteStr and ByteString from their text-oriented peers: ByteStr is a slice of raw bytes rather than necessarily valid UTF-8 text, while ByteString is a String-like type that can hold arbitrary binary data.
  Why it matters: The aliases make it easier to recognize when ActionKV handles raw bytes rather than guaranteed text.
• End-of-file detection: Explain that EOF is not a marker stored in the file: a read reaches EOF when the OS reports that zero bytes were read without an error.
  Why it matters: It helps distinguish reaching the end of stored data from finding a byte with a particular value.

Key terms: ActionKV::open(), ActionKV::load(), in-memory index, offsets, Result, ByteStr, ByteString, str, String, slices

7.7.2 Processing an individual record

The section explains how ActionKV reads a Bitcask record by using its fixed-size header to locate and check the key-value data.

• Bitcask record processing: Trace how ActionKV processes one record: it reads a 12-byte header containing a checksum, key length, and value length, then uses those values to read the remaining data and verify it.
  Why it matters: This explains how ActionKV can find and check an individual record on disk.

Key terms: Bitcask, record, process_record(), checksum, key_len, val_len

7.7.3 Writing multi-byte binary data to disk in a guaranteed byte order

This section explains why portable binary storage needs a defined byte order and shows how the byteorder crate supplies one when writing and reading multi-byte values.

• Portable byte order: Explain why storing multi-byte numbers requires a defined byte order when a file may be written on one computer and read on another.
  Why it matters: A fixed byte order helps another computer read stored numbers as intended.
• Byteorder read and write extensions: Describe how the byteorder crate provides explicit byte-order choices for reading and writing multi-byte values through Rust I/O types.
  Why it matters: These extensions let the key-value store use a predictable representation for binary values.

Key terms: multi-byte data, byte order, little endian, big endian, byteorder, std::io::Read, std::io::Write, ReadBytesExt, WriteBytesExt, LittleEndian

7.7.4 Validating I/O errors with checksums

Explains how checksums can detect corrupted disk data in actionkv, compares checksum choices, and illustrates parity checking.

• Checksum validation of disk data: Trace how a checksum is saved and checked when data is read, and explain what a mismatch means for actionkv.
  Why it matters: Detection lets actionkv alert the user to damaged data rather than silently accept it.
• Checksum function tradeoffs: Distinguish the checksum approaches compared here in terms of simplicity, speed, and confidence in their results.
  Why it matters: The appropriate checking function depends on what the application needs from error detection.
• Parity bit error checking: Explain how parity checking derives a bit from a stream of data and why a spare bit in an ASCII byte is convenient for it.
  Why it matters: Parity is a concrete example of a checking function applied to data bits.

Key terms: checksum, checking function, corrupted, parity bit, CRC32, cryptographic hash functions, parity checking, bitstream, ASCII

7.7.5 Inserting a new key-value pair into an existing database

This section explains how an append-only key-value store handles updates and deletions by appending records and letting later records supersede earlier ones during loading.

• Latest record precedence: Explain why update and delete can be implemented as variants of insert in this append-only database, and how loading the file makes newer records supersede stale data.
  Why it matters: It explains how the database can change a key’s effective value without rewriting an earlier record.

Key terms: append-only design, insert, update, delete, load(), stale data

7.7.6 The full code listing for actionkv

This subsection points readers to the full libactionkv code listing and marks the completion of a key-value store that stores and retrieves data.

7.7.7 Working with keys and values with HashMap and BTreeMap

The section distinguishes Rust’s key-value collections and explains how hashing enables lookup while collisions can slow it down.

• Key-value collection terminology: Distinguish the abstract idea of a collection of key-value pairs from Rust’s HashMap and BTreeMap implementations, and explain how the section uses the terms hash table and HashMap.
  Why it matters: This distinction helps you recognize when different names refer to the same kind of key-value collection rather than the same implementation.
• Hash functions and lookup: Explain how a hash function makes non-integer keys usable for lookup and why a hash map generally has constant-time lookup.
  Why it matters: It explains both the purpose of hashing and the usual lookup advantage of a HashMap.
• Hash collisions and slowdown: Describe why distinct keys can receive the same hash value and how handling those collisions can make lookup slower.
  Why it matters: Collisions explain why a hash map’s usual fast lookup has an important performance caveat.

Key terms: key-value pairs, associative array, hash table, HashMap, BTreeMap, hash function, hash value, constant time lookup, O(1), hash collision

7.7.8 Creating a HashMap and populating it with values

This subsection introduces populating a Rust HashMap and contrasts method calls with compile-time JSON literals.

• HashMap creation without literals: Explain that Rust's standard library has no HashMap literal syntax, so the example inserts key-value pairs and retrieves them using method calls.
  Why it matters: This explains why the basic HashMap example uses method calls rather than a built-in literal.
• Compile-time JSON literal alternative: Describe how the serde-json crate allows JSON literals in Rust source as a less verbose alternative to writing everything as method calls, with conversion preferably done at compile time to avoid runtime performance loss.
  Why it matters: It shows why the section offers a JSON-based alternative and why conversion timing matters.

Key terms: HashMap, literal syntax, method calls, serde-json crate, JSON string literals, compile time, runtime performance

7.7.9 Retrieving values from HashMap and BTreeMap

This section compares two ways to retrieve map values and identifies other HashMap operations and a reason to use BTreeMap.

• Map value retrieval choices: Distinguish square-bracket indexing from HashMap’s .get() method when retrieving a value, including what each returns and what happens if the key is missing.
  Why it matters: The choice determines whether a missing key panics or can be handled.
• Map operations and subset access: Describe the listed HashMap operations for removal and iteration, and state why the section points to BTreeMap for iterating over a subset.
  Why it matters: These operations show what HashMap offers and where this section draws a boundary for subset iteration.

Key terms: square brackets, &V, .get(), Option<&V>, .remove(), .keys(), .values(), .iter(), BTreeMap

7.7.10 How to decide between HashMap and BTreeMap

Gives a default rule for choosing between HashMap and BTreeMap and explains when BTreeMap may be preferable.

• HashMap and BTreeMap choice: Explain why HashMap is the default choice and when BTreeMap may be preferable: the keys have a natural order and the application uses that order.
  Why it matters: It gives you a practical default while identifying when key order matters to the choice.

Key terms: HashMap, BTreeMap, natural ordering

7.7.11 Adding a database index to actionkv v2.0

The section explains how actionkv v2.0 persists its index to avoid rebuilding it on every run and uses that index to locate records on disk.

• Index Rebuilding and Startup Time: Explain why actionkv v1 has poor startup time and how persisting its index addresses that problem.
  Why it matters: The index explains both the startup problem and the purpose of the v2.0 change.
• Persisted Index Record Lookup: Describe how a get operation uses the persisted index to find a key’s location on disk.
  Why it matters: Persisting an index is useful only if the store can load and use it to locate records.

Key terms: actionkv v1, actionkv v2.0, index, get operation, INDEX_KEY, on-disk, in-memory

7.8 Summary