Retention policy
A retention policy is a rule specifying what data or records are preserved, for how long, and under what conditions they may be archived, deleted, or otherwise disposed of.
Architecting an Apache Iceberg Lakehouse
A retention policy specifies how long table states and associated files or metadata may be retained before expiration or deletion.
- How it is used
- Controls Iceberg storage growth, historical access, secure deletion, and governance through coordinated snapshot and file lifecycle rules.
- How deep it goes
- Explains snapshot expiration, orphan cleanup, write-mode effects, references, and orchestration while deferring full workload-specific policy design.
- Examples
- Uses a seven-day Airflow expiration DAG, snapshot references, tags, branches, and COW or MOR file-lifecycle scenarios.
- What you can do after
- Operate/configure Iceberg lifecycle jobs that expire snapshots, distinguish orphan cleanup, and preserve required recovery or governance references.
Read how Architecting an Apache Iceberg Lakehouse explains retention policy →
Kafka for Architects
A retention policy is a rule governing how long or under what conditions stored Kafka records remain available.
- How it is used
- Governs topic history available for consumer recovery and replay, independently of whether records have been processed.
- How deep it goes
- Explains time, size, timestamps, and topic configuration, but leaves storage-segment internals to an external resource.
- Examples
- Uses a five-day dashboard topic configuration and a purchase-notification traffic surge to illustrate policy choices.
- What you can do after
- Operate/configure it with retention.ms, choosing a window that accommodates consumer disconnections and storage needs.
100 SQL Server Mistakes and How to Avoid Them
A data retention policy specifies how long stored data must be preserved before it may be deleted or archived.
- How it is used
- Uses retention duration to forecast database, backup, and storage requirements before capacity becomes an operational problem.
- How deep it goes
- Calculates linear data growth and associated storage needs, but stops short of defining deletion or archival procedures.
- Examples
- Illustrates retention through MarketingArchive, 5 GB daily ingestion, three-year retention, and air-gapped or immutable backups.
- What you can do after
- Evaluate storage forecasts and backup safeguards using retention duration, growth rate, headroom, and protection requirements.
Read how 100 SQL Server Mistakes and How to Avoid Them explains retention policy →
MongoDB 8.0 in Action, Third Edition
A retention policy defines how long backup copies or recovery data are preserved.
- How it is used
- Sets how long Atlas backup snapshots remain available to meet recovery and compliance needs.
- How deep it goes
- Shows schedule output with separate frequency and retention columns, but does not supply a command for changing retention settings.
- Examples
- Displays an Atlas CLI schedule with hourly through yearly snapshots and their retention values.
- What you can do after
- Operate/configure it by inspecting an Atlas backup schedule and distinguishing snapshot retention from a point-in-time recovery window.
Read how MongoDB 8.0 in Action, Third Edition explains retention policy →
Effective Data Analysis
A data retention policy specifies how long data may be kept and the conditions under which it must be anonymized or deleted.
- How it is used
- Uses retention limits to minimize stored personal data and support responsible analytics and privacy compliance.
- How deep it goes
- Explains retention periods and post-period actions, but does not specify detailed schedules, systems, or deletion procedures.
- Examples
- Uses e-commerce behavior data, anonymized user IDs, generalized locations, and a one-year retention example.
- What you can do after
- Operate/configure retention rules by setting periods and selecting deletion, scrubbing, anonymization, or access restrictions.
Read how Effective Data Analysis explains retention policy →
Apache Kafka in Action
A retention period is the duration for which Kafka retains records before allowing them to be deleted.
- How it is used
- Controls Kafka data availability, partition growth, retrieval windows, and historical-storage cost.
- How deep it goes
- Explains time- and size-based triggers, segment deletion, configuration, and partition effects without detailing broader retrieval implementation.
- Examples
- Demonstrates retention.ms=60000 on a topic and inspects the resulting partition files and segment replacement.
- What you can do after
- Operate/configure it by setting retention.ms or retention.bytes and checking segment deletion after records expire.
Just Use Postgres!
A data retention policy is a rule that automatically removes time-series chunks containing data older than a specified age.
- How it is used
- Automates removal of expired time-series chunks from hypertables and continuous aggregates.
- How deep it goes
- Defines retention intervals and cleanup schedules, but defers broader administration and lifecycle design.
- Examples
- Uses 30-day heart-rate measurement retention and seven-day low-heart-rate aggregate retention.
- What you can do after
- Operate/configure TimescaleDB cleanup with add_retention_policy and adjustable schedule intervals.
Elasticsearch in Action, Second Edition
A snapshot retention policy specifies the conditions and duration governing how long snapshots remain available.
- How it is used
- Automates snapshot expiry within scheduled Elasticsearch backup policies.
- How deep it goes
- Defines retention as a duration and demonstrates a seven-day JSON setting, without detailing expiration internals.
- Examples
- Uses midnight backups of movie and review indexes with date-based names and seven-day expiration.
- What you can do after
- Operate/configure retention through the Elasticsearch _slm API and a policy JSON request.
Read how Elasticsearch in Action, Second Edition explains retention policy →
Think Like a CTO
A retention policy specifies how long data must be kept and when or how it may be deleted or archived.
- How it is used
- Treats retention as a data-management lens for deciding how long each data bucket is kept and when it changes location or is deleted.
- How deep it goes
- Explains durations across operational stores and backups, storage aging, and deletion triggers without specifying exact periods.
- Examples
- Uses a client leaving as a business event that may trigger deletion, including from backups.
- What you can do after
- Operate data handling by assigning retention periods, aging rules, and deletion triggers to data buckets.
Build AI-Enhanced Web Apps
A retention policy specifies how long stored backups or other records are preserved before deletion.
- How it is used
- Governs storage duration and deletion for chat histories, backups, and other records within data protection and deployment operations.
- How deep it goes
- Explains retention decisions, deletion obligations, user communication, and backup practice, but does not provide implementation code or scheduling details.
- Examples
- Uses chat histories and daily backups with short retention as concrete policy applications.
- What you can do after
- Operate/configure a retention approach by setting periods, communicating them, and permanently deleting expired records.
Read how Build AI-Enhanced Web Apps explains retention policy →
Related concepts
- training datasetconfigures · A collection of examples or observations supplied to a learning procedure to fit or adapt a model or learn its patterns or parameters.
- logconfigures · A log is a record of events, activity, or diagnostic information produced during the operation of a system or process, often ordered by time.
- GDPRconfigured by · The General Data Protection Regulation (GDPR) is a European Union regulation governing the processing of personal data and protecting individuals’ rights concerning that data.
- signalconfigures · A signal is an observable value, event, or pattern that conveys information about a condition, state, or phenomenon.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.