Chapter 10: Maintaining an Iceberg lakehouse

Introduction

This introduction explains why an Iceberg lakehouse needs proactive maintenance after deployment.

• Need for proactive maintenance: Explain why maintaining an Iceberg lakehouse is an ongoing engineering responsibility and what can happen when maintenance is neglected.
  Why it matters: Deploying the lakehouse is not enough to keep it efficient and governed as it changes.

Key terms: proactive maintenance, data and metadata layout, storage footprint

10.1 Problem: Suboptimal data files

This section introduces why an Iceberg lakehouse needs ongoing data-file maintenance as its tables change.

• Changing data file layouts: Explain why an Iceberg lakehouse can become less efficient over time even when it was carefully designed.
  Why it matters: A table can keep working while gradually becoming slower and harder to operate.

Key terms: data files, query performance, metadata overhead, system stability

10.1.1 Small files

This section explains why small Iceberg data files can accumulate, how they affect query planning and execution, and why they need maintenance.

• Small File Origins: Explain how frequent streaming or micro-batch commits can produce many small Parquet files even though Iceberg handles the writes correctly, and why freshness requirements may make those commits necessary.
  Why it matters: It helps you distinguish a legitimate freshness tradeoff from a maintenance problem.
• File Count Execution Costs: Explain why high file counts can slow queries through file enumeration, object-store calls, and read scheduling rather than raw I/O bandwidth, and distinguish the HDFS block-size concern from cloud object storage.
  Why it matters: It explains why ample storage bandwidth does not necessarily make a high file count harmless.
• Metadata Growth And Maintenance: Explain how tracking each file makes Iceberg metadata grow as file counts rise, and why consolidating files through compaction helps control that persistent planning burden.
  Why it matters: It shows why small files call for ongoing maintenance, not just faster query execution.

Key terms: small files, streaming, micro-batch ingestion, frequent commits, freshness requirements, HDFS block size, cloud object storage, file count, object store API calls, read scheduling

10.1.2 Poorly colocated data

This section explains how poor data colocation slows queries and how Iceberg tables can be reorganized as access patterns change.

• Colocation and query scans: Explain why scattering related records across files makes queries read unnecessary data, especially when partitioning does not match common filters.
  Why it matters: Physical layout determines how much irrelevant data a query must read.
• Within-partition data locality: Describe how poorly ordered records within a partition can impair compression and data skipping, and how colocation can deteriorate over time.
  Why it matters: Choosing partitions once does not guarantee that data stays well organized.
• Adaptive physical data layout: Explain how partition evolution, sort orders, and compaction help align stored data with changing query patterns.
  Why it matters: Data layout needs to keep pace with the queries a lakehouse actually serves.

Key terms: colocation, partitioning, prune irrelevant files, query patterns, data locality, column-level compression, file or row-group statistics, Merge-on-Read (MOR), partition evolution, sort orders

10.1.3 Metadata sprawl

This section explains how Iceberg’s manifest and snapshot metadata can become a performance bottleneck and why maintenance must address both manifest fragmentation and snapshot history.

• Layered Iceberg metadata structure: Describe how snapshots, manifest lists, manifest files, and data files relate, and explain why Iceberg uses reusable manifests.
  Why it matters: The metadata layers explain both Iceberg’s snapshot consistency and where maintenance overhead arises.
• Manifest fragmentation from frequent commits: Explain how frequent small commits can create too many manifests and why that slows query planning even when data files are well laid out.
  Why it matters: Well-sized data files do not guarantee fast planning if their manifests are fragmented.
• Growing snapshot history costs: Explain how accumulating snapshots enlarges metadata.json and affects queries, commits, and snapshot expiration.
  Why it matters: Snapshot history can become a bottleneck even after data files and manifests have been consolidated.
• Coordinated metadata maintenance: Distinguish the roles of limiting unnecessary commits, rewriting manifests, and expiring snapshots in controlling metadata sprawl.
  Why it matters: Matching each maintenance action to its source of metadata growth helps preserve predictable performance.

Key terms: snapshot, manifest list, manifest file, data file, metadata sprawl, manifest files, query planning latency, snapshot history, metadata.json, snapshot expiration

10.1.4 Merge-on-Read performance hits

This section explains why Merge-on-Read can slow Iceberg queries, how its costs compare with Copy-on-Write, and how compaction restores read performance.

• Delete file read costs: Distinguish position deletes from equality deletes and explain why equality deletes can be more costly at read time.
  Why it matters: The kind of delete file affects how much work a query must do.
• MOR and COW tradeoffs: Explain how Merge-on-Read and Copy-on-Write distribute update costs between writes and reads, and relate the choice to a workload's update density and query patterns.
  Why it matters: Choosing a write mode means balancing ingestion speed against query performance and maintenance cost.
• Version-specific deletion overhead: Distinguish accumulating delete files in Iceberg v2 from deletion vectors in Iceberg v3.
  Why it matters: The maintenance problem described for accumulating v2 delete files should not be assumed to work the same way in v3.
• Compaction and degradation signals: Explain how compaction addresses MOR read overhead and describe signs and targeting considerations for maintenance.
  Why it matters: Compaction helps preserve MOR's write efficiency without allowing delete-related read costs to grow unchecked.

Key terms: position deletes, equality deletes, Merge-on-Read, MOR, Copy-on-Write, COW, update density, Iceberg v2, Iceberg v3, delete files

10.2 Solution: Compaction

Compaction reorganizes Iceberg data and metadata to improve performance, but file rewrites require careful scheduling and resource choices.

• Compaction and table layout: Explain how compaction reorganizes Iceberg data and metadata, and why the resulting layout helps queries.
  Why it matters: It connects the maintenance operation to the performance problems it is meant to fix.
• Compaction resource tradeoffs: Explain why compaction must be run thoughtfully rather than indiscriminately.
  Why it matters: A performance improvement can still harm the lakehouse if its resource demands interfere with other work.

Key terms: compaction, data files, delete files, manifest files, data colocation, SLAs, throughput constraints

10.2.1 What is compaction?

This section explains how Iceberg compaction rewrites data and metadata to keep tables efficient, and how to choose and scope compaction work.

• Compaction purpose and effects: Explain that compaction rewrites how Iceberg data is physically stored to improve reads and reduce metadata overhead, and describe the kinds of changes it can make.
  Why it matters: It connects routine table maintenance to the read-performance problems it is meant to solve.
• Bin-pack and sort strategies: Distinguish default bin-pack compaction from sort compaction, including why sorting or Z-ordering may help filtered queries and why Z-ordering is not a default choice.
  Why it matters: Choosing a strategy depends on whether file consolidation alone is enough or data locality is worth the extra work.
• Compaction procedures and scope: Describe the data-file and manifest rewrite procedures, their Spark and Dremio interfaces, and how options or filters limit the files or partitions rewritten.
  Why it matters: Selective rewrites let maintenance target useful improvements within available cluster capacity and SLA requirements.

Key terms: compaction, metadata overhead, predicate pushdown, delete files, bin-pack, sort compaction, Z-ordering, data locality, pruning, rewrite_data_files

10.2.2 Target file size

This section explains how to choose and validate compaction target file sizes while accounting for row-group behavior and workload constraints.

• File and row group sizing: Explain why compaction must consider both output file size and row-group size, and how poor sizing affects query execution.
  Why it matters: A larger output file is not necessarily a faster file to query.
• Workload dependent target sizes: Describe how the default target file size is configured and why an appropriate compaction target varies by workload.
  Why it matters: Treating the default as a starting point lets you balance file overhead against rewrite and query costs.
• Performance based size validation: Explain how to evaluate and refine a compaction file-size choice using observed query behavior.
  Why it matters: The useful target is the one that improves real workloads, not merely the one that produces the intended file sizes.

Key terms: compaction, row groups, parallelism, write.target-file-size-bytes, target file size, compression, table width, Iceberg metadata tables, query execution plans, I/O throughput

10.2.3 Files to be included

This section explains how Iceberg compaction selects and groups files, and how to tune that work to balance optimization against operational cost.

• File eligibility and grouping: Explain that compaction first selects eligible files, then groups them into independently processed units of work. Distinguish file-size thresholds, which affect eligibility, from min-input-files, which controls whether a formed group is compacted.
  Why it matters: This distinction helps you tune compaction without mistaking a group-size control for a file-selection rule.
• File size threshold tradeoffs: Explain how narrower versus wider ranges between minimum and maximum file sizes change the pool of eligible files and the cost of compaction.
  Why it matters: Threshold choices determine whether compaction is thorough enough without consuming too much time or compute.
• File group rewrite order: Describe how rewrite-job-order prioritizes file groups according to an operational goal.
  Why it matters: Ordering lets a maintenance job tackle the most useful groups first.
• Incremental compaction planning: Explain why frequent, targeted compaction jobs can maintain a table without disruptive full-table rewrites, and name the factors that guide file inclusion.
  Why it matters: Targeted, recurring work can preserve performance while limiting disruption to ingestion and other workloads.

Key terms: eligible files, file groups, max-file-group-size-bytes, min-input-files, min-file-size-bytes, max-file-size-bytes, compaction, rewrite-job-order, incremental compaction, metadata table queries

10.2.4 Using filters to scope compaction

This section explains how filters limit compaction to selected data so maintenance can be incremental, responsive to table health, and suited to individual workloads.

• Filtered compaction scope: Explain how Spark and Dremio filters restrict a compaction job to selected data, and why that is useful for large tables or shared infrastructure.
  Why it matters: Scoping a rewrite makes compaction more practical when a full-table job would be too disruptive or costly.
• Health-driven compaction targets: Describe how table health signals can guide a compaction job toward a particular segment instead of prompting an indiscriminate rewrite.
  Why it matters: Targeting the segment showing problems puts maintenance effort where it is needed.
• Workload-specific optimization choices: Explain why scoped compaction need not use the same optimization strategy for every table.
  Why it matters: Choosing optimization work by workload helps improve performance where it matters most.

Key terms: scoped compaction, SQL-like predicates, where parameter, FOR PARTITIONS clause, adaptive compaction, hot partitions, small files, delete files, compaction logic, sorting

10.3 Storage footprint management and data retention

Explains why Iceberg’s historical snapshots create a storage-retention tradeoff and make regular cleanup necessary.

• Snapshot retention tradeoff: Explain how time travel and snapshot isolation affect an Iceberg table’s storage footprint, and why regular cleanup matters.
  Why it matters: Historical access is useful, but retaining the files that support it has ongoing storage and performance costs.

Key terms: time travel, snapshot isolation, storage footprint, logically deleted, unreferenced files

10.3.1 Running snapshot expiration

Explains how snapshot expiration controls Iceberg storage and metadata growth, and how to set retention policies without losing needed history or disrupting queries.

• Snapshot expiration and file safety: Explain what snapshot expiration removes and why it does not delete files needed by active or retained snapshots.
  Why it matters: This distinction explains how cleanup can reduce storage without damaging the table’s retained states.
• Expiration commands and controls: Describe how Spark and Dremio run snapshot expiration and how a time boundary and minimum recent-snapshot count control retention.
  Why it matters: These controls let teams make expiration a repeatable maintenance task rather than an indiscriminate cleanup.
• Retention and running-query tradeoffs: Explain the consequences of expiring historical snapshots for time travel and long-running queries, and why schedules must reflect governance and workload needs.
  Why it matters: A cleanup schedule that saves space can still remove needed history or interrupt a query.
• High-frequency snapshot retention: Explain why CDC and near-real-time tables need retention tuned to commit frequency, and how time windows and tags can preserve useful history.
  Why it matters: High-ingestion tables need to control metadata growth without discarding the particular recovery points the organization values.

Key terms: snapshot expiration, retained snapshot, metadata files, data files, expire_snapshots, VACUUM TABLE, time boundary, retain_last, time travel, snapshot isolation

10.3.2 COW vs. MOR: Implications for data retention

Explains how copy-on-write and merge-on-read affect file retention, and how to coordinate snapshot expiration, compaction, ingestion, and tags to manage an Iceberg table.

• Copy-on-write file retention: Explain how a copy-on-write update or delete creates replacement data files and why the old files remain until their referencing snapshots expire.
  Why it matters: This explains why deleting rows does not necessarily free storage immediately.
• Merge-on-read delete handling: Describe how merge-on-read handles sparse changes without immediately rewriting whole data files, including how readers apply Iceberg v2 deletes and how v3 deletion vectors differ.
  Why it matters: The deferred rewrite explains MOR’s write advantage and its read and maintenance costs.
• Merge-on-read retention coupling: Explain why delete artifacts can keep MOR files and metadata around despite snapshot expiration, and distinguish the particular challenge of equality deletes.
  Why it matters: Expiring some snapshots may not reclaim the storage an operator expects.
• Snapshot expiration policy constraints: Explain how to set snapshot expiration in light of write frequency and query runtime, and why expiration by itself is insufficient to control the storage footprint.
  Why it matters: An overly aggressive policy can break running queries, while an expiration schedule alone may not shrink storage.
• Compaction scheduling and health signals: Describe how to schedule compaction around ingestion and interpret file, delete-artifact, and metadata trends between compaction cycles.
  Why it matters: Coordinating maintenance with writes avoids collisions and makes growth trends more useful than isolated counts.
• Tags as retention anchors: Explain how tags and branches affect cleanup and how to use tags selectively on fast-moving tables.
  Why it matters: A snapshot kept for audit or rollback can also keep an expensive file layout in storage.

Key terms: COW, snapshot, data files, time travel, MOR, position deletes, equality deletes, deletion vectors, base data file, delete artifacts

10.3.3 Regulatory considerations for data deletion

This section explains how Iceberg’s snapshots and deferred file deletion affect regulatory erasure, and what maintenance and retention practices make deletion complete.

• Logical versus physical deletion: Explain why deleting records from an Iceberg table does not immediately erase their data from storage, and why that matters for privacy requests.
  Why it matters: A successful delete operation alone does not prove that a person’s data has been erased.
• Complete deletion workflow: Describe how snapshot expiration, compaction, and orphan file cleanup work together to remove regulated data, including deleted rows in merge-on-read tables.
  Why it matters: Each maintenance step addresses a different reason deleted data may remain in storage.
• Retained snapshot references: Explain why deletion must account for every snapshot referencing the original files and for snapshots kept alive by branches or tags.
  Why it matters: An overlooked snapshot, branch, or tag can keep an old file—and its personal data—accessible.
• Compliance-aligned retention cadence: Describe how teams should schedule and oversee deletion maintenance so time travel and rollback do not exceed permitted retention windows.
  Why it matters: Maintenance that eventually runs may still be too late to satisfy a deletion deadline.

Key terms: logical deletion, physical removal, snapshots, compaction, MOR, delete files, expire_snapshots, rewrite_position_delete_files, orphan file cleanup, shared data files

10.4 Exploring Apache Iceberg’s metadata tables

This section explains how Iceberg metadata tables expose table state through SQL and help engineers identify maintenance needs.

• SQL visibility into table state: Explain that Iceberg exposes metadata through special tables that can be queried with standard SQL to inspect an Iceberg table’s internal state.
  Why it matters: You can inspect table state with familiar SQL rather than treating Iceberg metadata as a black box.
• Metadata-guided table maintenance: Describe the kinds of operational conditions metadata tables reveal and how engineers can use queries to prompt maintenance, including through monitoring pipelines or scheduled jobs.
  Why it matters: Table metadata can turn emerging performance and storage problems into visible maintenance signals.

Key terms: metadata tables, standard SQL queries, file-level insights, delete file tracking, snapshot growth, compaction, snapshot expiration

10.5 Summary