liveBook · Manning

Compaction

The consolidation or reorganization of data, objects, or context to reduce size or fragmentation while preserving necessary information or state.

How 10 Manning books explain it

Architecting an Apache Iceberg Lakehouse

A compaction strategy defines how fragmented data and delete files are consolidated based on factors such as activity, size, or freshness.

How it is used
Maintains Iceberg tables after frequent writes, updates, or deletes by controlling file fragmentation, metadata growth, and read overhead.
How deep it goes
Explains rewriting, bin-pack and sort strategies, file selection, filters, target sizing, and orchestration while deferring detailed specification semantics.
Examples
Uses Spark and Dremio rewrite procedures, bin-pack and sort commands, target file sizes, and region- and date-filtered maintenance jobs.
What you can do after
Operate/configure it through Spark, Dremio, or orchestration tools using thresholds, target sizes, sort strategies, predicates, and metadata-based triggers.

Read how Architecting an Apache Iceberg Lakehouse explains compaction →

Retrieval Augmented Generation, The Foundational Ideas

Context compression reduces retrieved or prompt context while retaining information relevant to the query and generation task.

How it is used
Reduces long or noisy retrieved context before generation to limit missed evidence, input cost, and latency.
How deep it goes
Works through document filtering, reordering, token pruning, and output recovery, then gives a baseline comparison and ratio-selection checklist.
Examples
Uses a legal assistant whose relevant clause was overlooked, then contrasts a successful 4× setting with a damaging 6× setting.
What you can do after
Operate/configure a compression pipeline by testing answer quality against an uncompressed baseline and backing off when critical details disappear.

Read how Retrieval Augmented Generation, The Foundational Ideas explains compaction →

ScyllaDB in Action

Compaction is a storage-maintenance process that merges multiple immutable database files into fewer, more compact files while reconciling their contents.

How it is used
Maintains SSTable storage by reducing redundant data so reads examine less data.
How deep it goes
Explains SSTable merging, tombstone timing, strategies, and operational monitoring, while deferring detailed strategy configuration.
Examples
Illustrates repeated author-bio updates scattered across SSTables and shows their consolidation into current data.
What you can do after
Diagnose/debug with nodetool compactionstats and evaluate major-compaction performance trade-offs.

Read how ScyllaDB in Action explains compaction →

Vibe Engineering

Automatic context compaction is the automatic summarization of an AI conversation's accumulated history to reduce retained context.

How it is used
Manages conversation history in long AI coding sessions while limiting retained context and repeated token processing.
How deep it goes
Explains when summarization occurs, what may be lost, and how to checkpoint information, without specifying the summarization algorithm.
Examples
Uses an append-only Findings.md and a ninety-minute compact-or-restart team rule to illustrate checkpoints.
What you can do after
Operate/configure Claude Code compaction with /compact and protect critical guidance through project files and timely notes.

Read how Vibe Engineering explains compaction →

Context Engineering

A process that summarizes an agent conversation and discards lengthy details while preserving high-level task information.

How it is used
Maintains continuity in long agent sessions by condensing relevant history within a finite context window.
How deep it goes
Explains triggering, retained state, drift mitigation, and fact-preservation checks, with an implementation example for compression.
Examples
Uses a return-policy ticket and a coding session with modified files and command outcomes to illustrate retained state.
What you can do after
Apply context compression with LLMLingua and check retained facts using assertions.

Read how Context Engineering explains compaction →

Spec-Driven Development

The process of replacing earlier interaction content with a shorter summary when an AI agent approaches its context limit.

How it is used
Identifies why coding-agent conversations can lose alignment and motivates moving important intent into durable artifacts.
How deep it goes
Defines the trigger and summary-based continuation, then traces lost constraints to observable failures without specifying a summarization algorithm.
Examples
Uses an early-session constraint lost from a summary and a new-session request for 90-day URL expiration.
What you can do after
Diagnose/debug with it by recognizing lost decisions and preserving critical constraints in artifacts that can be loaded into later sessions.

Read how Spec-Driven Development explains compaction →

.NET in Action, Second Edition

Garbage-collection compaction moves live objects together to reduce fragmentation and create larger contiguous free-memory regions.

How it is used
Explains why garbage collection may move arrays and why pointer-based C# code pins them.
How deep it goes
Works through allocation, collection, and compaction on a 1 MB heap, then relates movement to `fixed`; does not detail collector internals.
Examples
Uses a 1 MB heap allocation sequence, a heap-after-compaction figure, and a pointer-based array copy.
What you can do after
Diagnose/debug with it why free heap space may not satisfy an allocation and why pointer-based array access requires `fixed`.

Read how .NET in Action, Second Edition explains compaction →

Build an AI Agent (From Scratch)

A compaction strategy replaces processed or externally stored content in an agent's context with lightweight references that permit later retrieval.

How it is used
Controls growing agent context by replacing processed or externally stored tool data with references for later retrieval.
How deep it goes
Implements tool-specific rules and a single-pass scan, then places compaction in a token-threshold workflow with summarization fallback.
Examples
Uses a saved report, file reads, and web searches to demonstrate replaceable tool arguments and results.
What you can do after
Build it from scratch using tool-specific compaction rules and configure a token threshold for applying them.

Read how Build an AI Agent (From Scratch) explains compaction →

Build AI-Enhanced Web Apps

Prompt compression is the reduction of a prompt's token length while preserving information needed for the language model's task.

How it is used
Reduces oversized document-summarization prompts before LLM processing to address token limits, latency, and cost.
How deep it goes
Defines an intermediary compression step, outlines a small local LLM approach, and reports ratios, but gives no runnable compression implementation.
Examples
Compares standard prompts, conversation histories, and repetitive content using reported compression ratios.
What you can do after
Understand how an intermediary compressor can reduce prompt tokens and improve LLM turnaround time.

Read how Build AI-Enhanced Web Apps explains compaction →

The Art of AI Product Development

Prompt compression is the reduction of prompt context and examples while preserving information needed for high-quality model outputs.

How it is used
Reduces prompt token consumption and costs while supporting complex prompt-driven tasks.
How deep it goes
Defines compression as summarizing context and examples while preserving their essence; gives no specific procedure.
What you can do after
Apply it by summarizing context and examples to use fewer tokens while aiming to maintain high-quality outputs.

Read how The Art of AI Product Development explains compaction →

Also covered in

What to know first

Related concepts

Questions about compaction

What is compaction?

The consolidation or reorganization of data, objects, or context to reduce size or fragmentation while preserving necessary information or state.

Which Manning books explain compaction?

Architecting an Apache Iceberg Lakehouse, Retrieval Augmented Generation, The Foundational Ideas, ScyllaDB in Action, Vibe Engineering, Context Engineering, Spec-Driven Development, .NET in Action, Second Edition, Build an AI Agent (From Scratch) and 2 more.

What should I know before learning compaction?

Manning books that teach compaction usually introduce context window, AI agent, language model and reranking first.

What is the difference between compaction and deletion?

The consolidation or reorganization of data, objects, or context to reduce size or fragmentation while preserving necessary information or state. Deletion is an operation or request that removes an item, data, or resource from a collection or system.

What kinds of compaction are there?

Kinds of compaction include tool result compression.

What is compaction part of?

Compaction is part of context management. Context management is the deliberate control of information or resources available and maintained within an execution or interaction.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.