6 Handling CPU bound work

This chapter covers

The multiprocessing library
Creating process pools to handle CPU bound work
Using async and await to manage CPU bound work
Solving a map-reduce problem with asyncio
Handling shared data between multiple processes with locks
Improving the performance of work with both CPU and I/O bound operations

Until now, we’ve been focused on performance gains we can get with asyncio when running I/O bound work concurrently. Running I/O bound work is asyncio’s bread and butter, and with the way we’ve written code so far, we need to be careful not to run any CPU bound code in our coroutines. This seems like it severely limits asyncio, but the library is more versatile than just handling I/O bound work. Asyncio has an API for interoperating with Python’s multiprocessing library. This lets us use async await syntax as well as asyncio APIs with multiple processes. Using this, we can get the benefits of the asyncio library even when using CPU bound code. This allows us to achieve performance gains for CPU intensive work, such as mathematical computations or data processing, letting us sidestep the global interpreter lock and take full advantage of a multicore machine.

6.1 Introducing the multiprocessing library

6.2 Using process pools

6.2.1 Using asynchronous results

6.3 Using process pool executors with asyncio

6.3.1 Introducing process pool executors

6.3.2 Process pool executors with the asyncio event loop

6.4 Solving a map reduce problem with asyncio

6.4.1 A simple map reduce example

6.4.2 The Google Books Ngram Dataset

6.4.3 Mapping and reducing with asyncio

6.5 Shared data and locks

6.5.1 Sharing data and race conditions

6.5.2 Synchronizing with locks

6.5.3 Sharing data with process pools

6.6 Multiple processes, multiple event loops

6.7 Summary