Parallel processing executes separate portions of a computational task simultaneously. Distributed computing frameworks divide massive datasets into manageable tasks, process them across cluster nodes, and recombine the results, reducing runtime compared with a single machine. section 11.2.3
Working with parallel processing may require analysts to:
- Use a distributed framework’s SQL dialect, such as Spark SQL, when querying data. section 11.2.3
- Adopt distributed data-wrangling and modeling libraries, such as PySpark, instead of local tools. section 11.2.3
- Manage resource limits, node counts, query sizes, and processing costs. section 11.2.3
Figure 11.10 Parallel processing divides a data processing task into manageable chunks and recombines it into the final output.
Shows how parallel processing divides a data task into manageable chunks and recombines them into a final output. section 11.2.3
