liveBook · Manning

Dataset split

As explained in Grokking Concurrency

Partitioning is a parallel algorithm design technique that identifies independent computational tasks or data subsets for concurrent execution.

Partitioning is the first step in Foster’s methodology for designing concurrent systems. It breaks a problem into small tasks that can run independently, revealing opportunities for parallel execution. section 13.2 section 13.4

Two complementary ways to divide work are:

  • Data decomposition splits a large dataset into parts that can be processed independently. section 13.4
  • Task decomposition divides calculations by their function. A design can combine both approaches. section 13.4

Planning a road trip

Route planning, booking accommodation, and researching places to visit are separate tasks that friends can take on concurrently. The analogy makes independent work easier to spot before considering how to coordinate it. section 13.2

At this stage, look for as many small tasks as possible without accounting for processor count or machine type. The guidance is to find at least an order of magnitude more tasks than processors. Some algorithms cannot be decomposed; vertical scaling may help, but has physical limits. section 13.4 section 13.4

How Grokking Concurrency covers it

How deep it goes
Defines fine-grained task discovery and data or task decomposition, while deferring processor and machine constraints.
Examples
Uses road-trip planning tasks and independently processed data subsets as illustrations.
What you can do after
Build concurrent designs by identifying fine-grained tasks that can execute independently.
How it connects
Connects Foster’s design methodology with data decomposition, task decomposition, and vertical scaling.

Also explained in

Compare how 21 books explain dataset split →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Grokking Concurrency. The section links point to the parts of the book it draws on.