AWS Glue is a managed AWS service for data integration and metadata management. Its Spark-based ETL engine supports batch or event-driven pipelines that write curated data to Apache Iceberg tables on Amazon S3, while the Glue Data Catalog provides centralized table metadata. section 6.4.9 section 7.4.8
A typical Iceberg deployment uses Glue in several connected roles:
- Run Spark-based Glue jobs to ingest, transform, and write batch or streaming-oriented data into Iceberg tables. section 6.4.9
- Register and manage Iceberg table metadata through the regional AWS Glue Data Catalog. section 7.4.8
- Connect AWS-native services such as Athena, EMR, Redshift Spectrum, SageMaker Unified Studio, and Lake Formation to shared metadata and governance. section 7.4.8
- Apply fine-grained access control with Lake Formation and IAM, and audit metadata and schema changes with CloudTrail. section 7.4.8
Configuring an Iceberg Glue job
Enable native Iceberg support with --datalake-formats iceberg, select org.apache.iceberg.aws.glue.GlueCatalog as the catalog implementation, and use org.apache.iceberg.aws.s3.S3FileIO for S3-optimized I/O. Glue Spark Engine 3.0 and 4.0 include native transactional-table support; custom JARs or connectors can override the defaults when specific Iceberg versions or advanced features are required. section 7.4.8
Glue favors low operational overhead and tight AWS integration, but it couples metadata management to the AWS ecosystem. Its catalog does not natively version table metadata, so point-in-time recovery requires exported catalog entries and tracked metadata locations. section 7.5.2 section 11.3.1
