fsspec provides one Python interface for local filesystems, cloud storage, network protocols, archives, and filesystem-like objects. It reduces backend-specific code while preserving important behavioral differences, especially between local and remote storage. section 8.0 section 8.1
Use the common filesystem-style operations to:
- List directories and traverse trees with methods such as ls and walk. section 8.1.1
- Find files inside archives through ZipFileSystem without learning the zipfile-specific interface. section 8.1.2
- Replace a remote backend with a local filesystem while retaining the surrounding processing code. section 8.1.5
- Bridge storage into PyArrow and use Arrow filesystem primitives on top of an fsspec filesystem. section 8.1.6
Reading a CSV through nested filesystems
URL chaining addresses a file inside an archive, optionally inside a remote repository: fsspec.open("zip://dummy1.csv::github://tiagoantao:python-performance@/08-persistence/sec1-fsspec/dummy.zip"). The caller opens the resulting stream without explicitly downloading or opening the outer archive. section 8.1.4
from fsspec.implementations.zip import ZipFileSystem
def describe_all_csvs_in_zips(fs):
print(zip_name)
for zip_name in get_zips(fs):
my_zip = ZipFileSystem("/tmp/dl.zip")
for fname in my_zip.find(""): #1
if not fname.endswith(".csv"):
continue
print(fname)
df = pd.read_csv(my_zip.open(fname)) #2
print(df.describe())
Inspect the compact ZipFileSystem implementation that finds CSV files and passes archive streams directly to pandas. section 8.1.2
from pyarrow import csv
from pyarrow.fs import PyFileSystem, FSSpecHandler
zfs = ZipFileSystem("/tmp/dl.zip")
arrow_fs = PyFileSystem(FSSpecHandler(zfs))
my_csv = csv.read_csv(arrow_fs.open_input_stream("dummy1.csv"))
Inspect the adapter that exposes a ZipFileSystem to PyArrow through FSSpecHandler. section 8.1.6
