Merge-on-read (MOR) records updates and deletes separately from base data, then applies those changes during reads. It reduces write amplification and supports fine-grained corrections without rewriting entire tables, but shifts work into query execution and maintenance. section 6.3.1 section 10.1.4
Choose the delete representation according to writer capabilities and workload requirements:
- Use position deletes to identify rows by data-file position; they are generally faster to resolve and easier to compact, but require precise row identification. section 6.3.1
- Use equality deletes to match rows through column-value predicates; they suit streaming and CDC-style ingestion but cost more at read time and are harder to compact. section 6.3.1 section 10.1.4
- Verify engine support before choosing a strategy: Apache Spark supports position deletes, while Apache Flink supports equality deletes. section 6.3.1
Airbyte batch ingestion
Configure Airbyte’s S3 Data Lake connector to write Parquet data to Iceberg using MOR. The connector performs deduplication through equality deletes and inserts based on source primary keys and cursor ordering, so inconsistent or out-of-order source changes can undermine the strategy. section 6.4.6
MOR suits workloads that modify a small fraction of existing data or require high-frequency updates with low write amplification. Each read scans base files and applies relevant delete files; accumulated deletes, especially equality deletes, increase query complexity and can become a bottleneck. section 10.1.4 section 10.3.2
Figure 10.6 In Copy-on-Write, when you delete records, the data file with those records is rewritten without them. In Merge-on-Read, instead of rewriting a whole file, the deleted records are tracked in a delete file or deletion vector. This can be used to reconcile the table when it is read, which enables faster writing to the table but slows down reading.
Compare how copy-on-write rewrites affected data files while merge-on-read tracks changes in delete files or deletion vectors. section 10.1.4
Operate MOR tables with proactive maintenance. Monitor delete-file counts and sizes, trigger compaction when thresholds are exceeded, and use delete-file rewrites to reduce redundant entries. For secure deletion, coordinate compaction, snapshot expiration, and orphan cleanup because logical deletion does not immediately remove physical data. section 11.1.2 section 11.2.5
