chapter two

2 Extracting maximum performance from built-in features

 

This chapter covers

  • Profiling code to find speed and memory bottlenecks
  • Making more efficient use of existing Python data structures
  • Understanding Python’s memory cost of allocating typical data structures
  • Using lazy programming techniques to process large amounts of data

There are many tools and libraries to help us write more efficient Python. But before we dive into all the external options to improve performance, let’s first take a closer look at how we can write pure Python code that is more efficient, in both computing and IO performance. Indeed many, though certainly not all, Python performance problems can be solved by being more mindful of Python’s limits and capabilities.

2.1 Profiling applications with both IO and computing workloads

2.1.1 Downloading data and computing minimum temperatures

2.1.2 Using Python’s built-in profiling module

2.1.3 Using Local caches to reduce network usage

2.2 Profiling code to detect performance bottlenecks

2.2.1 Visualizing profiling information

2.2.2 Line profiling

2.3 Optimizing basic data structures for speed: lists, sets, dictionaries

2.3.1 Performance of list searches

2.3.2 Searching using sets

2.3.3 List, set and dictionary complexity in Python

2.4 Finding excessive memory allocation

2.4.2 The memory footprint of some alternative representations

2.4.3 Using arrays as a compact representation alternative to lists

2.4.4 Systematizing what we have learned: Estimating the memory usage of Python objects

2.5 Using laziness and generators for big-data pipelining