7 Files and storage

Storing data permanently on digital media is trickier than it looks. This chapter takes you though some of the details. To transfer information held by ephemeral electrical charges in RAM to (semi)permanent storage media and then be able to retrieve it again later takes several layers of software indirection.

The chapter introduces some new concepts such as how to structure projects into library crates for Rust developers. This task is needed because one of the projects is ambitious. By the end of the chapter, you’ll have built a working key-value store that’s guaranteed to be durable to hardware failure at any stage. During the chapter, we’ll work through a small number of side quests. For example, we implement parity bit checking and explore what it means to hash a value. To start with, however, let’s see if we can create patterns from the raw byte sequence within files.

7.1 What is a file format?

File formats are standards for working with data as an single, ordered sequence of bytes. Storage media like hard disk drives work faster when reading or writing large blocks of data in serial. This contrasts with in-memory data structures, where data layout has less of an impact.

7.2 Creating your own file formats for data storage

7.2.1 Writing data to disk with serde and the bincode format

7.3 Implementing a hexdump clone

7.4 File operations in Rust

7.4.1 Opening a file in Rust and controlling its file mode

7.4.2 Interacting with the filesystem in a type-safe manner with std::fs::Path

7.5 Implementing a key-value store with a log-structured, append-only storage architecture

7.5.1 The key-value model

7.5.2 Introducing actionkv v1: An in-memory key-value store with a command-line interface

7.6 Actionkv v1: The front-end code

7.6.1 Tailoring what is compiled with conditional compilation

7.7 Understanding the core of actionkv: The libactionkv crate