A callback is a user-supplied function invoked by another operation at a defined point. In asynchronous MapReduce, callbacks report job progress; their required arguments follow the reporting function’s protocol. section 3.3.2
Progress callbacks differ in when they run:
- A futures-based implementation calls the callback periodically while map or reduce jobs are running, passing a tag and counts of completed and remaining tasks. section 3.3.2
- The multiprocessing pool’s map_async callback runs only when all results are ready; per-item progress requires a lower-level approach that checks individual results. section 3.4.3
- Keep callbacks fast: execution waits for them. A callback’s return value could also be used to support cancellation. section 3.3.2
def reporter(tag, done, not_done):
print(f'Operation {tag}: {done}/{done+not_done}')
A minimal reporter prints the operation tag and completed-task count. section 3.3.2
