liveBook · Manning

Partition generator

As explained in Python Concurrency with asyncio

A partition generator is a generator that yields successive chunks of a larger sequence or collection.

A partition generator yields successive chunks from a larger sequence. It prepares data for parallel map operations without constructing every partition at once. section 6.4.3

Use partition size as a performance trade-off:

  • Choose chunks that are neither too large nor too small. section 6.4.3
  • Avoid tiny chunks because pickling and unpickling each partition can consume the gains from parallelism. section 6.4.3
  • Pass each generated chunk to a worker, then partition intermediate results again when parallelizing reduction. section 6.4.3
def partition(data: List,
              chunk_size: int) -> List:
    for i in range(0, len(data), chunk_size):
        yield data[i:i + chunk_size]

Shows the compact generator implementation, which advances through the input by chunk size and yields slices. section 6.4.3

How Python Concurrency with asyncio covers it

How deep it goes
Defines chunk generation and explains serialization costs, but defers broader partition-size optimization.
Examples
Uses word-frequency counting over Google Books data, including parallel map and reduce stages.
What it assumes
Assumes familiarity with sequences, slicing, generators, worker processes, and asynchronous execution.
What you can do after
Apply it to divide sequence data for worker processes and staged reduction.
How it connects
Links generators with multiprocessing, asyncio, process pools, and MapReduce.
Cover of Python Concurrency with asyncio
liveBookmanning.com
Read Python Concurrency with asyncio on liveBook Get the book at manning.com

This explainer was generated by Manning's knowledge graph from the text of Python Concurrency with asyncio. The section links point to the parts of the book it draws on.