2 Extracting maximum performance from built-in features

 

This chapter covers

  • Profiling code to find speed and memory bottlenecks
  • Making more efficient use of existing Python data structures
  • Understanding Python’s memory cost of allocating typical data structures
  • Using lazy programming techniques to process large amounts of data

There are many tools and libraries to help us write more efficient Python. But before we dive into all the external options to improve performance, let’s first take a closer look at how we can write pure Python code that is more efficient, in both computing and IO performance. Indeed many, although certainly not all, Python performance problems can be solved by being more mindful of Python’s limits and capabilities.

2.1 Profiling applications with both IO and computing workloads

2.1.1 Downloading data and computing minimum temperatures

2.1.2 Python’s built-in profiling module

2.1.3 Using local caches to reduce network usage

2.2 Profiling code to detect performance bottlenecks

2.2.1 Visualizing profiling information

2.2.2 Line profiling

2.2.3 The takeaway: Profiling code

2.3 Optimizing basic data structures for speed: Lists, sets, and dictionaries

2.3.1 Performance of list searches

2.3.2 Searching using sets

2.3.3 List, set, and dictionary complexity in Python

2.4 Finding excessive memory allocation

2.4.1 Navigating the minefield of Python memory estimation

2.4.2 The memory footprint of some alternative representations

2.4.3 Using arrays as a compact representation alternative to lists

2.4.4 Systematizing what we have learned: Estimating memory usage of Python objects

2.4.5 The takeaway: Estimating memory usage of Python objects