A text chunk is a bounded part of a larger input processed as one unit. In a word-counting pipeline, it can be a batch of words sent to a worker; in CSV processing, it can be a batch of parsed rows. The boundary is chosen for processing, not for sentence or topic meaning. section 3.4.4 section 8.3.2
Two implementations make the boundary explicit:
- For an iterator of words, collect up to chunk_size items and yield the batch. A worker applies the mapper to each word, then returns a list of results. The final batch may be smaller, and the caller must extend its results rather than append the returned list. section 3.4.4
- For a CSV file, pandas read_csv(chunksize=1000000) yields data frames of at most one million rows instead of one full-file data frame. Each can be converted to an Arrow table and written with ParquetWriter. section 8.3.2
def chunk(my_iter, chunk_size): #1
chunk_list = []
for elem in my_iter:
chunk_list.append(elem)
if len(chunk_list) == chunk_size:
yield chunk_list
chunk_list = []
if len(chunk_list) > 0:
yield chunk_list
def chunk_runner(fun, data): #2
ret = []
for datum in data:
ret.append(fun(datum))
return ret
def chunked_async_map(pool, mapper, data, chunk_size): #3
async_returns = []
for data_part in chunk(data, chunk_size): #4
async_returns.append(pool.apply_async( #5
chunk_runner, (mapper, data_part)))
return async_returns
The generator accepts an iterator, including a lazy one, and yields its last partial batch; chunk_runner processes each batch inside a pool worker. section 3.4.4
Measure the batch size
In a multiprocessing word-count test using Anna Karenina, batches of 1, 10, 100, 1,000, and 10,000 words took 114.2, 12.3, 4.3, 3.1, and 3.1 seconds respectively. Larger batches greatly reduced the cost of dispatching individual jobs, but the measured benefit leveled off. section 3.4.4
Chunk boundaries matter beyond text input. A ParquetWriter writes successive tables as separate row groups, which can later be read individually. Zarr stores array chunks separately and recommends processing chunk by chunk when possible, particularly for parallel writes. In Dask, work on array blocks can replace a task per element: a four-block example uses four initialization tasks instead of one million. These are related processing strategies, not rules for splitting prose. section 8.3.2 section 8.4.4 section 10.3.2
