liveBook · Manning

Object overhead

As explained in Fast Python: High performance techniques for large datasets

Object overhead is the memory required to represent an object beyond the memory occupied by its payload or explicitly stored data.

Object overhead is the memory required to represent an object beyond its payload or explicitly stored data. Python objects require at least 24 bytes of overhead, plus type-specific overhead, making many small objects disproportionately expensive. section 2.4.2

Object overhead becomes visible when comparing representations:

  • A small integer can occupy 28 bytes even when it represents one byte of useful data; a one-character string occupies 50 bytes. section 2.4.2
  • String storage varies with Unicode content: tested values range from 49 bytes for an empty string to 40,076 bytes for 10,000 emoji characters. section 2.4.2
  • Lists reserve extra capacity for future insertions, adding memory overhead that matters especially when many tiny lists are allocated. section 2.4.3
  • Homogeneous arrays store elements more compactly, while NumPy array size measurements separate raw data from Python, NumPy, and metadata overhead. section 2.4.3 section 4.1.1

Measuring payload versus object size

Use sys.getsizeof to measure an object's reported size, and use a NumPy array's nbytes to measure its raw element storage. A 45 × 182 uint8 image contains 8,190 bytes of data but reports 8,302 bytes as an array object, exposing 112 bytes of overhead and metadata. section 4.1.1

from sys import getsizeof
getsizeof('')
getsizeof('c')
getsizeof('c' * 10000)
getsizeof('ç' * 10000)
getsizeof('ç')
getsizeof('😐')
getsizeof('😐' * 10000)

Inspect the measured sizes of empty, ASCII, accented, and emoji strings to see how payload encoding and object overhead interact. section 2.4.2

How Fast Python: High performance techniques for large datasets covers it

How deep it goes
Explains baseline overhead, type-dependent allocation, spare list capacity, and NumPy metadata, but stops short of allocator internals.
Examples
Measures character strings, file contents, lists, byte arrays, and a grayscale NumPy image.
What it assumes
Assumes working knowledge of Python objects and containers, then demonstrates memory inspection with sys.getsizeof and NumPy nbytes.
What you can do after
Diagnose/debug with it by separating payload bytes from representation overhead when comparing Python containers and NumPy arrays.
How it connects
Connects Python objects with strings, integers, bytes, lists, arrays, NumPy metadata, views, and copies.
Cover of Fast Python: High performance techniques for large datasets
liveBookmanning.com
Read Fast Python: High performance techniques for large datasets on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Fast Python: High performance techniques for large datasets. The section links point to the parts of the book it draws on.