Compression
Compression is the transformation of information into a more compact representation to reduce storage or transmission requirements while retaining information needed for recovery or use.
LLMs in Production
Compression is the reduction of a model's size and computational requirements while retaining acceptable behavior and accuracy.
- How it is used
- Reduces model size to address deployment time, latency, GPU demand, and cost while preserving useful behavior.
- How deep it goes
- Explains quantization’s precision trade-off and illustrates post-training quantization and low-rank approximation without deriving their mathematics.
- Examples
- Uses a measurement analogy, decimal arithmetic, PyTorch post-training quantization, and an SVD matrix reconstruction.
- What you can do after
- Apply it via PyTorch post-training quantization, then assess whether reduced memory justifies any change in model behavior.
Sutskever's List
Compression is the representation of information using fewer bits or a shorter description while preserving the information needed for reconstruction or inference.
- How it is used
- Uses compression to expose regularity and select models that describe data without spending complexity on noise.
- How deep it goes
- Works through compressed-size proxies and MDL trade-offs, while noting that exact Kolmogorov complexity cannot be computed.
- Examples
- Contrasts repeated, random, and biased bit strings; compresses coarse-grained coffee arrays; and compares polynomial fits.
- What you can do after
- Evaluate/understand it: compare descriptions by length and distinguish compression of a network’s behavior from shrinking its parameters.
Fast Python: High performance techniques for large datasets
- How it is used
- Reduces storage needs and can accelerate array I/O when moving raw data costs more than decompressing it.
- How deep it goes
- Works through array write and read benchmarks, then compares compressors and bit shuffling; does not implement a compression algorithm.
- Examples
- Compares zero, repeated-tile, and random arrays under NumPy and Blosc, plus an xz-compressed taxi CSV loaded with pandas.
- What you can do after
- Apply it via Blosc or pandas, benchmarking time and size on representative data before choosing a compressor.
Read how Fast Python: High performance techniques for large datasets explains compression →
100 SQL Server Mistakes and How to Avoid Them
Data compression encodes stored database data in a more compact representation to reduce storage and I/O requirements.
- How it is used
- Uses compression as a performance enhancement for large tables when I/O limits performance and processor capacity remains available.
- How deep it goes
- Explains row-compression storage internals, demonstrates page reductions, and names page compression without detailing its internal structure.
- Examples
- Uses ImpressionArchive, page counts, ALTER TABLE rebuilds, metadata queries, and compression-savings estimation.
- What you can do after
- Operate/configure SQL Server compression, estimate savings, implement row or page compression, and verify resulting page usage.
Read how 100 SQL Server Mistakes and How to Avoid Them explains compression →
Kafka for Architects
Data compression is the transformation of data into a more compact representation to reduce storage or transmission size.
- How it is used
- Reduces Kafka network traffic and retained storage while informing message-size, batching, and cloud-cost decisions.
- How deep it goes
- Explains producer and broker placement, batch behavior, codecs, consumer decompression, and recompression overhead.
- Examples
- Uses gzip, snappy, lz4, zstd, uncompressed data, producer batches, and repetitive JSON or Avro payloads.
- What you can do after
- Operate/configure Kafka compression by selecting codecs, placement, batching, and limits against throughput, CPU, and storage requirements.
Classic Computer Science Problems in Swift: Essential techniques for practicing programmers
Compression is the process of representing data with fewer bits by eliminating or encoding redundancy.
- How it is used
- Frames compression as a practical way to reduce storage and bandwidth when space matters more than processing speed.
- How deep it goes
- Develops type narrowing and two-bit encoding with compression and decompression, but stops short of advanced pattern-based algorithms.
- Examples
- Uses oversized unsigned integers, a two-bit gene encoding, bit vectors, and internet file transmission as examples.
- What you can do after
- Apply compression by choosing compact numeric types and encoding fixed-alphabet data into bit vectors.
AI Agents in Action
Memory compression condenses stored memory information while retaining content useful for later retrieval.
- How it is used
- Condenses agent memories and knowledge stores to improve retrieval efficiency and relevance.
- How deep it goes
- Explains clustering, summarization, inspection criteria, and use-case variation, but does not specify the compression function’s internals.
- Examples
- Uses k-means clusters, Nexus compression interfaces, document stores, chunking experiments, and multiple memory-store types.
- What you can do after
- Apply compression in a knowledge or memory store and evaluate its effect on query or memory retrieval.
How Computers Make Books
Compression is the reduction of the amount of data needed to represent information while preserving either all or an acceptably useful part of its content.
- How it is used
- Reduces storage needs and transmission delays for text, images, and video.
- How deep it goes
- Explains reversible representations, pattern-based codes, and basic limits, while stopping short of specialized algorithms.
- Examples
- Uses a coffee-order abbreviation and a common-word code applied to a passage from Metamorphosis.
- What you can do after
- Evaluate simple compression schemes by comparing representations, repeated patterns, reversibility, and measured size.
AI Agents in Action, Second Edition
Compression is the reduction or summarization of stored information while preserving information needed for later retrieval or use.
- How it is used
- Uses compression to control token costs, reduce retrieval noise, and keep memory and knowledge stores scalable.
- How deep it goes
- Explains clustering, summarization, storage, triggers, and timing, but does not specify a compression-function implementation.
- Examples
- Uses clustered memories, verbose documents, repetitive code, and updated vector databases as concrete cases.
- What you can do after
- Apply it by clustering similar items, summarizing clusters, and scheduling compression according to store behavior.
Read how AI Agents in Action, Second Edition explains compression →
Logs and Telemetry
Compression is the encoding of data into a representation that requires less storage or transmission capacity.
- How it is used
- Reduces HTTP payload size when Fluent Bit sends substantial JSON data across networks.
- How deep it goes
- Explains enabling compression through a property and identifies gzip as the available algorithm, requiring support at both endpoints.
- Examples
- Demonstrates compression for Fluent Bit HTTP output carrying JSON data through data-center or proxy-connected networks.
- What you can do after
- Apply compression via Fluent Bit's HTTP output configuration using gzip.
WordPress Power Toolkit: Harness AI to build next-level websites
Compression is the reduction of digital data size while preserving the information needed to reconstruct or use it.
- How it is used
- Improves website delivery by shrinking video and code files, reducing bandwidth demands and helping pages load or play media more smoothly.
- How deep it goes
- Explains practical benefits and code minification, but does not detail compression algorithms or optimization settings.
- Examples
- Uses YouTube video embedding, WP Rocket minification, removed whitespace, and the compact notation “5bl” for repeated blank lines.
- What you can do after
- Apply it via YouTube and WP Rocket to reduce media or code size and improve website loading and playback.
Read how WordPress Power Toolkit: Harness AI to build next-level websites explains compression →
Related concepts
- Parquetused by · Apache Parquet is an open, column-oriented binary file format for storing and processing structured data with schema and metadata support.
- file sizeproduces · File size is the amount of digital storage space occupied by a file, typically measured in bytes or larger units.
- redundancyaddresses · Redundancy is the presence of duplicate or overlapping information, components, or resources, whether intentional or unnecessary.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.