Apache Flink is a high-performance, stateful stream-processing engine for building low-latency, event-driven ingestion pipelines into Apache Iceberg tables. It supports continuous data arrival, incremental commits, and both streaming and batch writes. section 6.4.2 section 2.7.2
Flink is most useful when ingestion requires:
- Exactly-once delivery through coordinated checkpointing and Iceberg atomic snapshots. section 6.4.2
- Event-time processing for out-of-order and late-arriving events, including watermarking and windowed aggregations. section 6.4.2 section 6.5.1
- In-flight filtering, enrichment, multistream joins, masking, deduplication, and field derivation through SQL or the DataStream API. section 6.4.2 section 6.5.3
- High-volume Kafka ingestion with checkpointing, partitioning, batching, and controlled commits. section 6.5.2
Streaming Kafka events into Iceberg
Define a Kafka source with an event-time watermark, configure an Iceberg sink, and insert the source stream into the sink. A typical design commits every few seconds to meet freshness targets while avoiding excessive snapshot and metadata growth. section 6.5.1
Flink trades operational simplicity for control. Continuous pipelines require tuning of checkpointing, memory, file sizes, partitioning, commit frequency, schema validation, compaction, and monitoring. Complex stateful transformations also demand stronger testing and operational maturity. section 6.5.1 section 6.5.3
CREATE TABLE source_kafka ( id STRING, event_time TIMESTAMP(3), data STRING, WATERMARK FOR event_time AS event_time - INTERVAL '5' SECOND ) WITH ( 'connector' = 'kafka', 'topic' = 'events', 'properties.bootstrap.servers' = 'broker:9092', 'format' = 'json' ); #1 CREATE TABLE sink_iceberg ( id STRING, event_time TIMESTAMP(3), data STRING ) WITH ( 'connector' = 'iceberg', 'catalog-name' = 'my_catalog', 'catalog-type' = 'hadoop', 'warehouse' = 's3a://my-bucket/warehouse', 'format-version' = '2' ); #2 INSERT INTO sink_iceberg SELECT * FROM source_kafka; #3
Inspect the executable SQL pattern for defining a Kafka source, applying event-time watermarks, configuring an Iceberg sink, and inserting the stream. section 6.4.2
