Chunk granularity sets how many array elements each Zarr chunk holds. Chunks partition large arrays into manageable units, and their shape can be specified when an array is created. section 8.4.2
Choose chunk size for the workload and storage system:
- Calibrate against the algorithms and access patterns; avoid chunks that are too small, and keep chunks comfortably in memory. section 8.4.3
- Try different values across dimensions: chunk shape affects performance and whether work fits in memory. section 8.4.3
- Account for the store: DirectoryStore can struggle with thousands of files in one directory; NestedDirectoryStore distributes chunks into subdirectories. section 8.4.3
A two-dimensional array
An array with shape (3,976,554, 210) uses chunks of (20,000, 210), grouping 20,000 rows while keeping all 210 columns together. This choice assumes all individuals will be read at once. section 8.4.3
Chunk shape : (41632,) Chunks initialized : 8/8
A one-dimensional Zarr array reports a chunk shape of (41,632,) and eight initialized chunks. section 8.4.2
