concept speculative execution in category data

appears as: speculative execution
Hadoop in Action

This is an excerpt from Manning's book Hadoop in Action.

Hadoop uses the idempotency property again to mitigate the slow-task problem. Instead of restarting a task only after it has failed, Hadoop will notice a slow-running task and schedule the same task to be run in another node in parallel. Idempotency guarantees the parallel task will generate the same output. Hadoop will monitor the parallel tasks. As soon as one finishes successfully, Hadoop will use its output and kill the other parallel tasks. This entire process is called speculative execution.

Note that speculative execution of map tasks will take place only after all map tasks have been scheduled to run, and only for map tasks that are making much less progress than is average on the other map tasks. It’s the same case for speculative execution of reduce tasks. Speculative execution does not “race” multiple copies of a task to get the best completion time. It only prevents the slow tasks from dragging down the job’s completion time.