liveBook · Manning

S3FileIO

As explained in Architecting an Apache Iceberg Lakehouse

S3FileIO is an Apache Iceberg file I/O implementation for reading and writing table data through S3-compatible object storage.

S3FileIO is Apache Iceberg’s file I/O implementation for storing and retrieving table data through Amazon S3 or S3-compatible object storage. In a Spark, Nessie, and MinIO deployment, it connects Iceberg table operations to a configured warehouse and storage endpoint while leaving catalog responsibilities to Nessie. section 3.3.3

Deployment choices determine how S3FileIO fits the lakehouse architecture:

  • Use it with SparkCatalog and Nessie settings to write Iceberg tables into MinIO, with branching, time travel, and atomic commits. section 3.3.3
  • Use org.apache.iceberg.aws.s3.S3FileIO in AWS Glue when native Iceberg support and optimized S3 I/O are required. section 7.4.8
conf = (
    pyspark.SparkConf()
        .setAppName('Iceberg Ingestion')
        .set('spark.jars.packages', 
             'org.postgresql:postgresql:42.7.3,'
             'org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.5.0,'
             'org.projectnessie.nessie-integrations:nessie-spark-extensions-3.5_2.12:0.77.1,'
             'software.amazon.awssdk:bundle:2.24.8,'
             'software.amazon.awssdk:url-connection-client:2.24.8')
        .set('spark.sql.extensions', 
             'org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions,'
             'org.projectnessie.spark.extensions.NessieSparkSessionExtensions')
        .set('spark.sql.catalog.nessie', 'org.apache.iceberg.spark.SparkCatalog')
        .set('spark.sql.catalog.nessie.uri', CATALOG_URI)
        .set('spark.sql.catalog.nessie.ref', 'main')
        .set('spark.sql.catalog.nessie.authentication.type', 'NONE')
        .set('spark.sql.catalog.nessie.catalog-impl', 'org.apache.iceberg.nessie.NessieCatalog')
        .set('spark.sql.catalog.nessie.s3.endpoint', STORAGE_URI)
        .set('spark.sql.catalog.nessie.warehouse', WAREHOUSE)
        .set('spark.sql.catalog.nessie.io-impl', 'org.apache.iceberg.aws.s3.S3FileIO')
) #1

Use the Spark configuration listing to assemble Iceberg, Nessie, AWS SDK, catalog, and S3-compatible storage settings. section 3.3.3

How Architecting an Apache Iceberg Lakehouse covers it

How deep it goes
Shows configuration placement and deployment context, but stops short of explaining S3FileIO internals.
Examples
Configures SparkCatalog against Nessie and MinIO, then selects S3FileIO in AWS Glue for S3-backed Iceberg tables.
What it assumes
Assumes familiarity with Iceberg table operations, Spark configuration, catalogs, and S3-compatible storage endpoints.
What you can do after
Operate/configure Iceberg storage through Spark or AWS Glue using S3-compatible endpoints and warehouse settings.
How it connects
Connects Apache Iceberg with Spark, Nessie, MinIO, Amazon S3, and AWS Glue.
Cover of Architecting an Apache Iceberg Lakehouse
liveBookmanning.com
Read Architecting an Apache Iceberg Lakehouse on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Architecting an Apache Iceberg Lakehouse. The section links point to the parts of the book it draws on.