3 Concurrency, parallelism and asynchronous processing in Python

Modern CPU architectures allow for more than one sequential program to be executed at the same time, permitting speed ups up to the number of parallel processing units available (e.g. CPU cores). However most Python code is normally sequential, so it is not able to use all available CPU resources. This means that most Python code doesn’t take advantage of modern hardware’s capabilities, and tends to run at a much lower speed than the hardware allow. So we need to devise techniques to get help Python make use of all the available CPU power. In this chapter we are going learn how to do just that, starting with some approaches you may be familiar with in general, but which have some unique Python twists. We will discuss concurrency, multi-threading and parallelism the Python way, including some strong limitations around multi-threaded programming.

3.1  Writing the scaffold of an asynchronous server

3.1.1  The implementation of the scaffold for communicating with clients

3.1.2  Programming with coroutines

3.1.3  Sending complex data from simple synchronous client

3.1.4  Alternative approaches

3.2  Implementing the first MapReduce engine

3.2.1  Understanding MapReduce frameworks

3.2.2  Developing a very simple test scenario

3.2.3  Implementing a too-simple MapReduce framework

3.3  Implementing a concurrent version of a MapReduce engine

3.3.1  Using concurrent.futures to implement a threaded server

3.3.2  Asynchronous execution with Futures

3.3.3  The GIL and multi-threading

3.4  Using multi-processing to implement MapReduce

3.4.1  A solution based on concurrent.futures

3.4.2  A solution based on the multiprocessing module

3.4.3  Monitoring the progress of the multiprocessing solution