In this chapter:
- Learn how data is represented on physical storage devices
- Write your own data structures to your preferred file format
- Build a tool to read from files and inspect their contents
- Create a working key-value store that’s immune from becoming corrupt
Storing data permanently on digital media is trickier than it looks. This chapter takes you though some of that detail. To transfer information held by ephemeral electrical charges in RAM to (semi-)permanent storage media—and then to be able to retrieve it again later—takes several layers of software indirection.
The chapter introduces some new concepts for Rust developers, such as how to structure projects into library crates. This is needed because one of the projects is ambitious. By the end of the chapter, you would have built a working key-value store that’s guaranteed to be durable to hardware failure at any stage.
During the chapter, we’ll work through a small number of side quests. For example, we implement parity bit checking and explore what it means "to hash" a value. To start with however, let’s see if we can create patterns from the raw byte sequence within files.
File formats are standards for working with data as an single, ordered sequence of bytes. Storage media, such as hard disk drives, work fastest when reading or writing large blocks of data in serial. This contrasts with in-memory data structures, where data layout has less of an impact.