A dataset split, or partition, is an independently accessible physical portion of a larger logical dataset. Partitioning assigns data to separate portions so reads and writes need not all contend for the same resources. It can also make a dataset practical to place across CPU cores or machines when copying the whole dataset to every location would be costly. section 5.1 section 5.0
Consider users whose records are divided by geographic location. Requests concerning users in New York can reach one partition, while requests concerning users in Sydney reach another. The separation can reduce synchronization between unrelated writes and allow parallel processing. Its benefit depends on routing each request to the right partition and keeping access reasonably balanced. section 5.1 section 5.0
Choose a split along two dimensions:
- Physical layout: horizontal partitioning distributes complete records—table rows, documents, or key–value pairs. Vertical and hybrid layouts are other options. section 5.2.1 section 5.7
- Workload boundary: user-based, geographic, functional, and time-based partitioning organize data around how it is accessed. These choices can be combined with a physical layout. section 5.3 section 5.7
Figure 5.1 Horizontally partitioning a database table. On the left, you can see the full logical table with four rows. On the right side, you can see three horizontal partitions.
The table diagram makes a horizontal split concrete: one logical table’s rows are placed in three physical partitions. section 5.2.1
Placement matters as well as membership. On a NUMA machine, an application can divide a key–value store’s state among nodes and steer each lookup toward a core near the responsible data. This reduces the need to access remote memory. Across machines, partitioning similarly provides separate places to handle requests, but requires a request-routing strategy. section 3.4 section 5.7
Watch for hot partitions. A zip-code range split may put far more users in some ranges than others; hashing the key can distribute those keys more evenly. Equal key counts do not guarantee equal partition sizes when values vary, as with stored images. Changing a poor scheme may require rebalancing partitions on a live system. section 5.5.1
Measure routing before adding data placement
A Rust REST API with embedded SQLite can assign each server a node ID and port, then use a partition-aware HTTP benchmark to send requests round-robin to one or five nodes. The five-node run has lower tail latency because more requests execute in parallel. This measures request routing and server parallelism, not a completed data split: placing data by partition would also require a partitioning strategy in the client. section 5.6
A useful split makes independently accessed data independently processable. Check both the latency gain and its costs: routing, uneven load, rebalancing, and operations that need to span partitions. section 5.0 section 5.2 section 5.5.1
