Blosc compresses binary data, including NumPy arrays. Compression can make storage access faster, not just smaller: when moving raw data leaves the CPU waiting on memory or storage, the work of decompressing a smaller representation may cost less than the time saved moving it. The result depends on how compressible the data is. section 6.2 section 6.2.1
import os import blosc import numpy as np random_arr = np.random.randint( 256, size=(1024, 1024, 1024)).astype(np.uint8) zero_arr = np.zeros(shape=(1024, 1024, 1024)).astype(np.uint8) rep_tile_arr = np.tile( np.arange(256).astype(np.uint8), 4*1024*1024).reshape(1024,1024,1024) def write_numpy(arr, prefix): np.save(f"{prefix}.npy", arr) #1 os.system("sync") #2 def write_blosc(arr, prefix, cname="lz4"): b_arr = blosc.pack_array(arr, cname=cname) #3 w = open(f"{prefix}.bl", "wb") w.write(b_arr) w.close() os.system("sync") #2 def read_numpy(prefix): return np.load(f"{prefix}.npy") def read_blosc(prefix): r = open(f"{prefix}.bl", "rb") b_arr = r.read() r.close() return blosc.unpack_array(b_arr) #4 #1 NumPy can take care of disk persistence natively. #2 sync forces the disk to flush. #3 If we want to write a NumPy array in Blosc, we need to pack it. #4 If we want to read a NumPy array in Blosc, we need to unpack it.
The Python helpers provide a direct comparison: NumPy saves and loads `.npy` files, while Blosc packs arrays before writing and unpacks them after reading. The three test arrays—zeros, a repeated tile, and random values—have sharply different compressibility. section 6.2.1
Measure both time and size before choosing compressed storage:
- For zero and repeated-tile arrays, Blosc writes took 0.53 seconds versus 7.49 seconds with NumPy. The resulting files were about 250 and 200 times smaller, respectively. section 6.2.1
- For random data, compression provided no size reduction; NumPy wrote faster, 7.5 versus 8.13 seconds. section 6.2.1
- Cold reads showed the same divide: Blosc took about 0.6 seconds for the compressible arrays, versus about 7 seconds with NumPy, but was slower for random data. section 6.2.2
Disk benchmarks need cache control. Flush pending writes when timing persistence, and avoid measuring reads served from operating-system buffers. The shown cache-dropping command requires root on Debian/Ubuntu derivatives; it is not portable to Windows, macOS, or every Linux distribution. section 6.2.1 section 6.2.2
Tune compression for the data
Pass `cname='lz4'` or `cname='zstd'` to `blosc.pack_array`. On the repeated-tile array, LZ4 took 527 ms and produced 5,204 KB; Zstandard took 919 ms but produced 366 KB. Blosc also accepts `shuffle=blosc.BITSHUFFLE`: in a separate comparison, shuffling reduced the output from 5,345,500 to 4,600,034 bytes while increasing compression time from 524 to 596 ms. Test the speed–size trade-off on your own data. section 6.2.3 section 6.2.4
