liveBook · Manning

Read-write skew

As explained in Machine Learning System Design

An imbalance or mismatch between the read and write workloads, volumes, or latency requirements of a feature management system.

Read-write skew is a feature-store design pattern in which read and write workloads differ in volume, timing, or latency requirements. Design starts by estimating operation loads, data volumes, critical read latency, and feature refresh frequency. section 11.4.2

Account for the main workload differences:

  • Write features in batches, streams, or mixed schedules rather than assuming one universal update pattern. section 11.4.2
  • Prioritize read latency and verify that the feature-store infrastructure meets its nonfunctional requirements. section 11.4.2
  • Combine precomputed features with real-time features during reads when recent events matter. section 11.4.2

Mixed update schedules

A long-running job may compute some features daily, while lightweight features are streamed in near real time. Both can be consumed through the same feature-store read path. section 11.4.2

How Machine Learning System Design covers it

How deep it goes
Defines the workload mismatch and lists design questions, but stops short of implementation details or capacity calculations.
Examples
Uses daily feature jobs, near-real-time streams, and combined precomputed and real-time reads as examples.
What you can do after
Build it from scratch by sizing reads and writes and selecting batch, streaming, or combined feature-update paths.
How it connects
Connects feature stores with batch processing, stream processing, precomputed features, real-time features, and nonfunctional requirements.
Cover of Machine Learning System Design
liveBookmanning.com
Read Machine Learning System Design on liveBook Get the book at manning.com

This explainer was generated by Manning's knowledge graph from the text of Machine Learning System Design. The section links point to the parts of the book it draws on.