Manifest rewriting

As explained in Architecting an Apache Iceberg Lakehouse

RewriteManifests is an Apache Iceberg maintenance procedure that rewrites manifest files to consolidate and compact table metadata.

Manifest rewriting is an Apache Iceberg maintenance procedure that consolidates fragmented manifest files describing table data. Fewer, better-organized manifests reduce metadata traversal during query planning and snapshot management without rewriting the underlying data files. section 10.1.3 section 10.2.1

Use manifest rewriting when metadata growth becomes a reliability, latency, or scalability concern:

• Run it alongside snapshot expiration; data-file compaction alone does not control growth in manifests and metadata.json. section 10.1.3
• Treat it as distinct from rewrite_data_files: manifest rewriting reorganizes metadata, while data-file compaction changes physical file layout. section 10.2.1
• Schedule it regularly or trigger it from metadata conditions; incremental maintenance can do little work or exit when optimization is unnecessary. section 11.1.2

Spark procedure

Invoke the Iceberg Spark procedure with: CALL catalog_name.system.rewrite_manifests('db.sales'); section 10.2.1

Figure 10.5 Ideally, you’ll have fewer manifest files tracking fewer data files containing a greater number of records per file, minimizing the cost of metadata traversal and data file I/O.

Use the ideal metadata layout as a visual target: fewer manifest files should track fewer, better-sized data files, reducing metadata traversal and data-file I/O. section 10.1.3

How Architecting an Apache Iceberg Lakehouse covers it

How deep it goes

Explains manifest consolidation, distinguishes it from data-file compaction, and stops short of detailed rewrite algorithms.

Examples

Uses the Spark rewrite_manifests procedure and contrasts fragmented metadata with an ideal compact layout.

What it assumes

Assumes familiarity with Iceberg tables, manifests, snapshots, and metadata tables while distinguishing manifest work from data-file compaction.

What you can do after

Operate/configure Iceberg maintenance by invoking rewrite_manifests and scheduling or triggering it when metadata fragmentation warrants work.

How it connects

Connects manifest rewriting to metadata sprawl, snapshots, snapshot expiration, query planning, and data-file compaction.

liveBookmanning.com

Read Architecting an Apache Iceberg Lakehouse on liveBook Get the book at manning.com

Related concepts

• compactionused by · Compaction is the consolidation or reorganization of data, objects, or context to reduce length or fragmentation while preserving what is needed.
• metadata sprawladdressed by · Metadata sprawl is the accumulation and fragmentation of table metadata that increases the cost of planning queries and managing table state.

This explainer was generated by Manning's knowledge graph from the text of Architecting an Apache Iceberg Lakehouse. The section links point to the parts of the book it draws on.