Change data capture processes records representing inserts, updates, and deletes from a source system. Streaming pipelines maintain state across events, making CDC practical without repeatedly scanning entire tables. In Iceberg lakehouses, pipelines commit changes in bounded units rather than writing individual records. section 6.2.3
Choose the write approach according to update workload, engine support, and read performance:
- Copy-on-write rewrites affected data files, avoiding delete files but increasing write amplification and update latency. section 6.3.1
- Merge-on-read records changes in delete files; equality deletes suit flexible CDC ingestion, while position deletes are faster to resolve and compact. section 6.3.1
- Apache Flink supports equality deletes, event-time processing, deduplication, late data, and exactly-once delivery through checkpointing and Iceberg snapshots. section 6.4.2
Figure 6.10 Merge transactions allow you to compare records to a staging dataset and conditionally insert, update, and delete records.
Use merge transactions to compare incoming records with staging data and conditionally insert, update, or delete rows. section 6.3.1
