liveBook · Manning

Zstandard

As explained in Fast Python: High performance techniques for large datasets

Zstandard, or ZSTD, is a general-purpose lossless compression algorithm and format with configurable trade-offs between compression ratio and speed.

Zstandard (ZSTD) is a compression option for trading storage efficiency against compression speed. It appears in both in-memory experiments and columnar Parquet storage, where smaller representations can also reduce storage and cache pressure. section 6.2.3 section 8.2.2

The measured and practical trade-offs are:

  • Compared with LZ4 through Blosc, Zstandard takes longer to compress but produces a substantially smaller representation. section 6.2.3
  • A benchmark compresses the same representation in 919 ms at 366 KB with Zstandard, versus 527 ms at 5,204 KB with LZ4. section 6.2.3
  • Applied to every column in the example Parquet file, ZSTD reduces storage from 110 MB with Snappy to 82 MB. section 8.2.2

Selecting Zstandard in Blosc

Use Blosc's array-packing interface with cname='zstd' to benchmark or create a Zstandard-compressed representation, then compare its timing and byte length with cname='lz4'. section 6.2.3

Writing ZSTD-compressed Parquet

Pass compression="ZSTD" to pq.write_table when writing a Parquet table to apply Zstandard compression across its columns. section 8.2.2

%timeit rep_lz4 = blosc.pack_array(rep_tile_arr,
  cname='lz4')                                          #1
rep_lz4 = blosc.pack_array(rep_tile_arr, cname='lz4')
%timeit rep_std = blosc.pack_array(rep_tile_arr,
  cname='zstd')                                         #2
rep_std = blosc.pack_array(rep_tile_arr, cname='zstd')
print(len(rep_lz4) // 1024)
print(len(rep_std) // 1024)

Inspect the runnable comparison that times Blosc packing with LZ4 and Zstandard and prints both compressed sizes. section 6.2.3

How Fast Python: High performance techniques for large datasets covers it

How deep it goes
Benchmarks timing and size against LZ4 and applies ZSTD to Parquet, but stops short of an exhaustive algorithm survey.
Examples
Uses Blosc array packing, an LZ4 comparison, Parquet writing, and an 110 MB-to-82 MB storage measurement.
What you can do after
Apply it via Blosc or PyArrow Parquet and evaluate compression time against resulting storage size.
How it connects
Connects compression choices with LZ4, Blosc, Parquet, Snappy, column encoding, dictionaries, and run-length encoding.
Cover of Fast Python: High performance techniques for large datasets
liveBookmanning.com
Read Fast Python: High performance techniques for large datasets on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Fast Python: High performance techniques for large datasets. The section links point to the parts of the book it draws on.