Compaction
Compaction is the consolidation or reorganization of data, objects, or context to reduce length or fragmentation while preserving what is needed.
Architecting an Apache Iceberg Lakehouse
A compaction strategy defines how fragmented data and delete files are consolidated based on factors such as activity, size, or freshness.
- How it is used
- Maintains Iceberg tables after frequent writes, updates, or deletes by controlling file fragmentation, metadata growth, and read overhead.
- How deep it goes
- Explains rewriting, bin-pack and sort strategies, file selection, filters, target sizing, and orchestration while deferring detailed specification semantics.
- Examples
- Uses Spark and Dremio rewrite procedures, bin-pack and sort commands, target file sizes, and region- and date-filtered maintenance jobs.
- What you can do after
- Operate/configure it through Spark, Dremio, or orchestration tools using thresholds, target sizes, sort strategies, predicates, and metadata-based triggers.
Read how Architecting an Apache Iceberg Lakehouse explains compaction →
ScyllaDB in Action
Compaction is a storage-maintenance process that merges multiple immutable database files into fewer, more compact files while reconciling their contents.
- How it is used
- Maintains SSTable storage by reducing redundant data so reads examine less data.
- How deep it goes
- Explains SSTable merging, tombstone timing, strategies, and operational monitoring, while deferring detailed strategy configuration.
- Examples
- Illustrates repeated author-bio updates scattered across SSTables and shows their consolidation into current data.
- What you can do after
- Diagnose/debug with nodetool compactionstats and evaluate major-compaction performance trade-offs.
.NET in Action, Second Edition
Garbage-collection compaction moves live objects together to reduce fragmentation and create larger contiguous free-memory regions.
- How it is used
- Explains why garbage collection may move arrays and why pointer-based C# code pins them.
- How deep it goes
- Works through allocation, collection, and compaction on a 1 MB heap, then relates movement to `fixed`; does not detail collector internals.
- Examples
- Uses a 1 MB heap allocation sequence, a heap-after-compaction figure, and a pointer-based array copy.
- What you can do after
- Diagnose/debug with it why free heap space may not satisfy an allocation and why pointer-based array access requires `fixed`.
Read how .NET in Action, Second Edition explains compaction →
Build an AI Agent (From Scratch)
A compaction strategy replaces processed or externally stored content in an agent's context with lightweight references that permit later retrieval.
- How it is used
- Reduces growing agent context by replacing processed tool data with references that support later retrieval.
- How deep it goes
- Implements tool-name rules for ToolCalls and ToolResults, then places compaction behind a token threshold; defers lengthy conversation history to summarization.
- Examples
- Uses a large report saved with create_file, file reads, web searches, and Python replacement rules.
- What you can do after
- Build it from scratch as a rule-based request transformation and place it before summarization in a threshold-driven optimizer.
Read how Build an AI Agent (From Scratch) explains compaction →
The Art of AI Product Development
Prompt compression is the reduction of prompt context and examples while preserving information needed for high-quality model outputs.
- How it is used
- Reduces prompt token consumption and costs while supporting complex prompt-driven tasks.
- How deep it goes
- Defines compression as summarizing context and examples while preserving their essence; gives no specific procedure.
- What you can do after
- Apply it by summarizing context and examples to use fewer tokens while aiming to maintain high-quality outputs.
Read how The Art of AI Product Development explains compaction →
Related concepts
- deletioncontrasts with · Deletion is an operation that removes an item, association, or resource from a collection or system, or requests or marks it for removal.
- summarizationimplemented by · Summarization is the process of condensing source content into a shorter representation that preserves its important information or meaning.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.