Zarr stores homogeneous, multidimensional arrays persistently while retaining a NumPy-style interface. Its groups organize arrays in a tree, and its stores can use different backends. For large arrays, storage choices matter: chunk size, compression, and filesystem layout affect how data is read and written. section 8.4 section 8.4.1
Inspect a chromosome dataset
Open `db.zarr` and inspect its tree. Each chromosome group contains `positions`, `alleles`, and `calls` arrays. The chromosome-1 calls matrix has 318,558 rows and 210 columns; its values encode the calls needed for a combined matrix for PCA. Arrays can be accessed by paths such as `genomes["chromosome-2/calls"]`. section 8.4.1
def traverse_hierarchy(group, location=""):
for name, array in group.arrays(): #1
print(f"{location}/{name} {array.shape} {array.dtype}")
for name, group in group.groups(): #2
my_root = f"{location}/{name}"
print(my_root + "/")
traverse_hierarchy(group, my_root)
traverse_hierarchy(genomes)
This traversal uses `arrays()` and `groups()` to inspect array shapes and types throughout the hierarchy. section 8.4.1
A Zarr array is partitioned into chunks. In the directory store shown here, each chunk is a separate file: a 333,056-element positions array has eight chunks of 41,632 elements. Its Blosc/LZ4-compressed storage is 687,723 bytes versus 2,664,448 uncompressed bytes. Zarr chose that chunk size when the array was created, but chunk size can be specified; choose it for the intended work rather than relying on the guess. section 8.4.2
Compressor : Blosc(cname='lz4', clevel=5, shuffle=SHUFFLE, blocksize=0) No. bytes : 2664448 (2.5M) No. bytes stored : 687723 (671.6K) Storage ratio : 3.9
The array metadata reports the compressor, byte counts, and 3.9 storage ratio used to assess this storage choice. section 8.4.2
To concatenate chromosome calls, a sequential implementation copies each chromosome array into slices of one output array. A parallel version assigns output chunks to multiprocessing workers, which read chromosome data and write separate chunks. Chunk-aligned writes avoid concurrent writes to the same chunk file. The sequential result occupies 283.3 MB for 796.4 MB of uncompressed data; no measured parallel speedup is given. Processing chunk by chunk is also suited to data too large to fit in memory. section 8.4.4
from functools import partial
from multiprocessing import Pool
partial_process_chunk = partial(
process_chunk, genomes,
all_calls, chrom_sizes, CHUNK_SIZE)
def do_parallel():
with Pool() as p:
p.map(partial_process_chunk, range(all_calls.nchunks))
do_parallel()
A multiprocessing pool maps one task per output chunk; the worker’s chunk-boundary logic is shown separately. section 8.4.4
