An S3-compatible API is an object-storage interface that follows Amazon S3 request and data-access conventions. It provides consistent, HTTP-based operations across cloud and on-premises object stores, separating storage access from vendor-specific infrastructure. section 5.3.2 section 5.3
The API uses buckets and unique object keys rather than hierarchical directories. Each object is independently addressable and treated as an immutable binary blob, allowing clients to retrieve multiple objects concurrently without filesystem locking. section 5.3.2
S3-compatible access supports Iceberg architectures by providing:
- Interoperability across query engines, ingestion frameworks, storage vendors, and deployment environments. section 4.4.1 section 5.3.2
- Independent reads and writes for Iceberg data files and metadata files stored as objects. section 5.3.2
- A storage model that fits Iceberg snapshot commits, where metadata is written before an updated pointer is committed. section 5.3.2
- Compatibility with operational tooling such as backup systems, monitoring dashboards, and access-control frameworks. section 5.3.2
Spark with MinIO and Iceberg
A PySpark pipeline can set WAREHOUSE to s3://warehouse/ and STORAGE_URI to a MinIO endpoint, then configure Spark with Iceberg and AWS SDK dependencies. Spark uses the S3 API to read and write Iceberg table data while Nessie tracks table versions. section 3.3.3
Figure 5.7 An engine running an Iceberg transaction uses the S3 API to talk to an object storage platform, where it stores data as Parquet files along with Iceberg metadata.
Trace the storage path from an engine running an Iceberg transaction through the S3 API to object storage containing Parquet data and Iceberg metadata. section 5.3
