An equality delete represents rows to remove by specifying column values that match records across data files. It is a merge-on-read mechanism: changes are stored in delete files and applied when the table is queried, avoiding immediate rewrites of the affected data files. section 6.3.1
Use equality deletes with these architectural tradeoffs in mind:
- Prefer them for flexible streaming and change-data-capture ingestion, where the writer may not know the physical file and row position. section 6.3.1
- Expect higher read cost because query engines evaluate delete predicates against data during execution. section 6.3.1 section 10.1.4
- Plan compaction and workload controls because accumulating equality delete files makes reconciliation harder and can degrade high-churn query performance. section 10.1.4
- Compare engine support before standardizing: the supplied discussion identifies Apache Flink as supporting equality deletes, while Apache Spark currently supports position deletes. section 6.3.1
CDC-driven correction
A streaming CDC pipeline can write equality deletes for records identified by logical column values, then write replacement records as new data files. Queries reconcile the base files, delete files, and replacements into the current view. section 6.3.1
Figure 6.9 When using delete files, updated and deleted records in existing data files are ignored, while new data files containing only the updated records are written.
Visualizes how delete files cause updated and deleted records in existing data files to be ignored while replacement records are written separately. section 6.3.1
