Merge operation

As explained in Architecting an Apache Iceberg Lakehouse

A merge operation conditionally inserts, updates, or deletes target records based on matches with a source dataset.

A merge operation compares target records with a staging dataset and conditionally inserts, updates, or deletes rows. It extends append-only ingestion for reconciliation, late-arriving data, and slowly changing dimensions. section 6.3.1 section 6.4.1

Use merge operations to:

• Match source and target records on a key, then update matched rows. section 6.4.1
• Insert source records without a target match. section 6.4.1
• Handle row-level changes through copy-on-write or merge-on-read strategies, balancing read performance against write amplification and latency. section 6.3.1

Spark SQL upsert

MERGE INTO my_catalog.db.my_table t USING my_catalog.db.staging_table s ON t.id = s.id WHEN MATCHED THEN UPDATE SET * WHEN NOT MATCHED THEN INSERT * section 6.4.1

Figure 6.10 Merge transactions allow you to compare records to a staging dataset and conditionally insert, update, and delete records.

Visualizes how a merge transaction compares records with staging data and applies conditional inserts, updates, and deletes. section 6.3.1

How Architecting an Apache Iceberg Lakehouse covers it

How deep it goes

Explains matching and conditional row actions with a working Spark example, but does not detail transaction internals.

Examples

Uses a keyed staging-table upsert with matched updates and unmatched inserts.

What it assumes

Assumes Iceberg and Spark familiarity sufficient to identify target and staging tables and configure Spark integration.

What you can do after

Apply it via Spark SQL to reconcile staging records with an Iceberg target table.

How it connects

Connects Iceberg with Spark SQL, staging datasets, copy-on-write, merge-on-read, and slowly changing dimensions.

liveBookmanning.com

Read Architecting an Apache Iceberg Lakehouse on liveBook Get the book at manning.com

Related concepts

• Sparkuses · Apache Spark is an open-source distributed computing engine for large-scale data processing and analytics across clusters, supporting batch and streaming workloads.
• merge clausecontrasts with · The Cypher MERGE clause matches a specified graph pattern and creates any parts of it that do not already exist.

This explainer was generated by Manning's knowledge graph from the text of Architecting an Apache Iceberg Lakehouse. The section links point to the parts of the book it draws on.