liveBook · Manning

Data skew

As explained in Latency

Data skew is an uneven distribution of records or work across partitions.

Data skew is an uneven distribution of data or workload across horizontal partitions. Some nodes may be underused while others are overused, limiting the benefit of distributing a database across nodes. section 5.2.1

A partitioning key with only a few distinct values can leave some partitions much larger than others. Cardinality—the number of unique values in a column—therefore matters when choosing the key. User IDs are an example of a high-cardinality key. section 5.2.1

Check data size and access patterns separately:

  • For uneven data distribution, choose a high-cardinality key; if skewed writes enlarge a few partitions, rebalance them. section 5.2.1
  • For hot partitions, partition further within them or give them more resources. section 5.2.1

How Latency covers it

How deep it goes
Explains causes and remedies through key cardinality, hot partitions, and skewed writes; provides no measurement procedure.
Examples
Uses user IDs as a high-cardinality column and hot partitions and skewed writes as imbalance scenarios.
What you can do after
Operate/configure it by selecting a high-cardinality key, splitting hot partitions, allocating resources, or rebalancing after skewed writes.
How it connects
Connects data skew to horizontal partitioning, partitioning keys, cardinality, and workload distribution.
Cover of Latency
liveBookmanning.com
Read Latency on liveBook Get the book at manning.com

This explainer was generated by Manning's knowledge graph from the text of Latency. The section links point to the parts of the book it draws on.