liveBookmanning.com

What you will learn in Architecting an Apache Iceberg Lakehouse

A scalable, open-source data platform

by Alex Merced
Foreword by Tim Berglund
Afterword by Adi Polak

Read it on liveBook Get the book at manning.com

Chapter by chapter

Concepts explained in this book

• Apache IcebergAn open table format for managing large analytic datasets with metadata-driven structure, snapshots, schema evolution, and transactional write capabilities.
• compactionA compaction strategy defines how fragmented data and delete files are consolidated based on factors such as activity, size, or freshness.
• snapshot expirationAn Apache Iceberg procedure that expires unneeded snapshots and removes data or metadata files no longer referenced by retained table states.
• Dremio CloudDremio is a data lakehouse platform that provides SQL query, semantic, and data-management capabilities over diverse data sources.
• merge-on-readMerge-on-read mode is a table write strategy that records changes separately and merges them with base data when data is read.
• semantic layerA semantic layer is a governed modeling layer that defines consistent business entities, dimensions, and metrics across data sources.
• SparkApache Spark is an open-source distributed processing framework for large-scale batch, streaming, and analytical workloads.
• object storeObject storage is a storage system that manages data as self-contained objects addressed through a service interface rather than a hierarchical filesystem.
• data lakehouseA data lakehouse is a data architecture that combines the openness and flexible storage of a data lake with the governance and analytical capabilities of a data warehouse.
• MinIOMinIO is an S3-compatible object-storage system that can be deployed on premises or in other infrastructure.
• lakehouse catalogA lakehouse architectural component that tracks and governs table metadata and enables data systems to discover and access tables.
• delete fileA delete file is an auxiliary data structure that records rows or positional entries to be excluded from table results.
• schema evolutionSchema evolution is the controlled modification of a dataset schema over time while preserving compatibility with existing data and readers.
• Project NessieProject Nessie is an open-source transactional catalog that provides Git-like branching and versioning for data and metadata.
• snapshotA consistent versioned representation of a table's state, including the data files that constitute that state.
• orphan file cleanupAn Apache Iceberg procedure that identifies and removes files in storage that are not referenced by table metadata.
• authorizationAccess control is the restriction of data or system operations to authorized users or services according to defined permissions.
• Iceberg REST CatalogThe Iceberg REST Catalog specification is a REST-based open standard for implementing catalogs compatible with Apache Iceberg.
• copy-on-writeCopy-on-write is a data-modification pattern that rewrites affected base files immediately to materialize updates or deletions.
• FlinkApache Flink is an open-source distributed processing framework for stateful batch and stream computations.
• retention policyA retention policy specifies how long table states and associated files or metadata may be retained before expiration or deletion.
• ParquetApache Parquet is an open columnar file format designed for efficient storage and analytical processing of large datasets.
• replicationData replication is the process of maintaining copies of data in multiple storage systems or locations.
• auditabilityAuditability is the extent to which data access, changes, and control operations can be recorded and examined for verification or compliance.
• requirementsA storage requirement is an explicit, testable condition that a storage system or architecture must satisfy.
• TrinoPrestoSQL was the forked project name that later became Trino.
• metadataMetadata is structured information that describes the location, structure, contents, versions, or other properties of data.
• deletion vectorA deletion vector is a data-file-associated structure that records rows to be treated as deleted without requiring a separate delete file for each deletion operation.
• Iceberg tableAn Iceberg table is a logical table whose data and metadata are managed using the Apache Iceberg table format.
• observabilityObservability is the capability to infer a system's internal behavior and resource usage from its emitted telemetry and operational data.
• partition pruningPartition pruning is the elimination of data partitions that cannot satisfy a query's filters before or during data access.
• metadata tableA metadata table is a queryable representation of metadata describing the state or contents of a data table.
• catalogA catalog is a service or system that records, governs, and exposes metadata about data assets such as tables, schemas, locations, and access permissions.
• Apache SupersetApache Superset is an open-source web application for data exploration, dashboarding, and business-intelligence visualization.
• data residencyData residency is the requirement that data be stored or processed within specified geographic or jurisdictional boundaries.
• role-based access controlRole-based access control is an authorization technique that grants permissions according to assigned user or service roles.
• query pushdownFilter pushdown is a query optimization that evaluates filtering conditions as close as possible to the source data.
• provenanceData lineage tracking is the practice of recording how data moves between sources, transformations, storage locations, and outputs.
• storage abstractionA storage layer is a lakehouse architectural layer that stores data on cost-effective storage systems.
• checkpointingA checkpoint is persisted progress information from which an incremental data-processing workflow can resume after interruption.
• throughputThe maximum volume or rate of data that an ingestion system can process within a specified period.
• metadata.jsonmetadata.json is an Apache Iceberg table metadata file that records table versions, schemas, partition specifications, and related table state.
• batch pipelineBatch ingestion is the process of loading accumulated data into a target system in discrete processing runs.
• query engineA query engine is a software system that parses, plans, and executes queries against data sources.
• open table formatA table format is a specification for representing a logical table and its metadata independently of the underlying physical data storage.
• hard deletePhysical deletion is the removal of deleted data from the files or objects that physically store it.
• logical deletionLogical deletion is a table-state change that excludes records from the current logical view without immediately removing their physical representations.
• hybrid storage modelA hybrid storage architecture combines multiple storage environments, such as cloud object storage and on-premises storage, in one data platform.
• Apache PolarisApache Polaris is an open-source catalog service for Apache Iceberg tables with governance and role-based access-control capabilities.
• data governanceData governance comprises the policies, processes, and controls for ensuring that data is managed, protected, documented, and used appropriately.
• consumption layerA lakehouse architectural layer that exposes data to business-intelligence, artificial-intelligence, and real-time applications.
• credential vendingStorage credential vending is the controlled issuance of temporary credentials for accessing storage resources.
• lifecycle policyData lifecycle management is the policy-based control of data placement, retention, deletion, archival, and tiering throughout its lifecycle.
• Iceberg branchAn Iceberg branch is a named reference to a table snapshot that supports independent versioned table-state progression.
• position deleteA position delete is an Iceberg delete record that identifies a row by its position within a data file.
• workload separationWorkload isolation is the separation of computational workloads so that resource use or failures in one workload do not adversely affect another.
• federated query engineA federated query engine is a query engine that executes queries across multiple heterogeneous data sources without requiring all data to be stored in one system.
• field idA stable field ID is an identifier that preserves a column's logical identity independently of its name or position in a schema.
• Apache ArrowApache Arrow is a software project and standardized in-memory columnar data format for efficient data interchange across systems and programming languages.
• row-level securityA row-level filter restricts access to records according to predicates applied at query or policy evaluation time.
• operational burdenOperational overhead is the additional effort, resources, and complexity required to deploy, operate, maintain, and support a system.
• interoperabilityInteroperability is the ability of different software systems or components to exchange data and operate together through compatible interfaces and conventions.
• manifest rewritingRewriteManifests is an Apache Iceberg maintenance procedure that rewrites manifest files to consolidate and compact table metadata.
• partition evolutionThe ability to change a table's partition strategy over time without rewriting existing data.
• rollbackAn operation that makes an earlier table snapshot the current state without necessarily deleting files introduced by later snapshots.
• Hadoop Distributed File SystemHadoop Distributed File System is a distributed file system designed to store very large datasets across clusters of commodity machines.
• AWS GlueAWS Glue is a managed cloud service for data integration, discovery, preparation, and movement across data sources and destinations.
• runtime filteringA query-execution technique that derives runtime filter values from one part of a query and applies them to reduce data scanned elsewhere.
• Azure Data Lake StorageAzure Data Lake is Microsoft's cloud storage and analytics environment for storing and processing large volumes of data.
• tagA snapshot tag is a named Iceberg reference that protects a specific table snapshot from expiration.
• NetApp StorageGRIDNetApp StorageGRID is an enterprise object-storage platform designed to manage S3-compatible data across distributed and hybrid environments.
• dashboardA dashboard is a visual interface that combines charts and other summaries for monitoring or analyzing data.
• hidden partitioningA partitioning method in which partition values are derived from transforms and are not exposed as query columns.
• MCPThe Model Context Protocol is a standardized protocol that lets large language models interact with external data, tools, and services through governed calls.
• WORM policyWORM is a storage property that permits data to be written once while preventing subsequent modification or deletion for the protected retention period.
• snapshot lineageSnapshot lineage is the history of relationships and changes among versioned table snapshots over time.
• business intelligence toolA BI tool is a business-intelligence software application for querying, analyzing, and visualizing organizational data.
• block storageBlock storage provides persistent storage as independently addressable fixed-size blocks that operating systems or applications assemble into volumes and filesystems.
• uptimeHigh availability is the capability of a system to remain operational with minimal interruption despite component or site failures.
• data integrityData integrity is the preservation of data's correctness, completeness, consistency, and protection from unauthorized alteration or loss.
• cachingCaching is the technique of storing reusable data or computation results in a faster-access location to reduce subsequent processing or retrieval cost.
• query planningQuery planning is the process of determining an execution strategy for a query, including which files or partitions must be read.
• S3 APIAn S3-compatible API is an object-storage interface that implements the request and data-access conventions of Amazon S3.
• object lockingAn object lock configuration applies retention or deletion restrictions to objects in storage.
• eventual consistencyEventual consistency is a distributed-storage consistency model in which replicas may temporarily differ but converge to the same value after updates propagate.
• ACIDThe transaction guarantees of atomicity, consistency, isolation, and durability that preserve reliable database state changes.
• PyIcebergA software development kit that reads and parses Apache Iceberg metadata files and exposes their contents programmatically.
• data lakeA data lake is a storage-centered data platform that retains data in flexible formats on relatively inexpensive storage for varied downstream processing.
• hybrid deploymentA hybrid cloud deployment combines infrastructure operated on premises with services or storage provided by public clouds.
• Apache AirflowApache Airflow is an open-source platform for authoring, scheduling, and monitoring workflows as directed acyclic graphs.
• exactly-once deliveryExactly-once semantics guarantee that each input record contributes its intended processing effect no more than once despite retries or failures.
• directed acyclic graphA directed acyclic graph is a graph of nodes and directed dependencies with no cycles, commonly representing workflow tasks and their execution order.
• row groupA row group is a storage and I/O unit containing a subset of a columnar file's rows.
• Dell ECSDell ECS is an enterprise object-storage platform that supports distributed storage, policy-based placement, replication, retention, and access controls.
• erasure encodingErasure encoding is a data-protection technique that reconstructs unavailable data from distributed fragments and parity fragments.
• equality deleteAn equality delete is a row-deletion representation that identifies records to remove by matching specified column values.
• self-managed catalogA self-managed deployment is an arrangement in which an organization installs, operates, and maintains the software in its own chosen infrastructure.
• manifest listA manifest list is an Apache Iceberg metadata structure that represents a snapshot by listing its manifest files and summary statistics.
• metadata catalogA data catalog is a system that stores and exposes metadata for discovering, describing, and governing data assets.
• table metadataTable metadata is information describing a table's schema, data files, versions, partitioning, and related structural state.
• TableauTableau is a business-intelligence and data-visualization software platform for exploring data and creating interactive reports and dashboards.
• resilienceFault tolerance is the ability of a system to continue operating correctly or recover without unacceptable data loss or corruption after component failures.
• data warehouseA data warehouse is a centralized repository optimized for analytical querying and reporting over integrated historical data.
• register_tableregister_table is an Iceberg procedure that creates a catalog entry pointing to an existing table metadata file.
• access control listAn access control list is a set of permissions attached to a resource that specifies which identities or groups may perform particular operations.
• ReflectionsReflections are Dremio-managed precomputed representations of source data or query results that can transparently substitute into query plans.
• snapshot-based architectureAn architecture that represents table state through immutable or versioned snapshots of metadata and data references.
• delete-file rewritingDelete-file rewriting is the process of rewriting delete files and affected data files to consolidate delete state and reduce query-time merge work.
• CephCeph is an open-source distributed storage platform that provides object, block, and file storage.
• Docker ComposeDocker Compose is a tool for defining and running multi-container applications from a declarative configuration file.
• cross-region recoveryCross-region failover is the process of switching data operations from a failed region to a replicated region.
• cross-table transactionA multitable transaction is an atomic operation whose coordinated changes span multiple tables and commit as one consistent unit.
• reproducibilityReproducibility is the ability to obtain consistent analytical or computational results by repeating a process with the same inputs, versions, and conditions.
• FivetranFivetran is a managed data-integration platform that automates movement of data from sources to destinations.
• metadata sprawlMetadata bloat is excessive growth of metadata that increases storage, catalog, or query-planning overhead.
• file-level metadataMetadata describing individual data files, including their locations and query-relevant statistics.
• metadata designThe organization and representation of metadata used to describe, locate, version, and plan access to table data.
• auditAuditing is the systematic examination and recording of data and system activities to support accountability, compliance, and operational assurance.
• RewriteDataFilesRewriteDataFiles is an Apache Iceberg maintenance procedure that rewrites table data files to consolidate or improve their physical layout.
• secure data deletionThe controlled removal of data and its retained representations so that unauthorized recovery is prevented.
• Cloud Columnar CacheA cloud columnar cache is a local disk-based cache that stores frequently accessed cloud data blocks in compressed columnar form.
• freshnessData freshness is the recency of data relative to the time it was generated, updated, or expected to be available.
• PostgreSQLPostgreSQL is an open-source object-relational database management system.
• ODBCODBC is a language-independent API standard for connecting applications to database management systems and executing SQL statements.
• idempotencyThe property that repeating an operation produces the same effective result as performing it once.
• computational notebookJupyter is an open-source interactive computing environment for combining executable code, narrative text, data, and visualizations.
• orphaned fileA physical data or metadata file that is no longer referenced by the table's valid metadata state.
• disaster recoveryDisaster recovery is the set of processes and capabilities for restoring data and system operation after failures, corruption, or other disruptive events.
• encryption at restEncryption at rest is the protection of stored data by encoding it so that it cannot be read without the required cryptographic key.
• backup and restoreA backup is a separate copy of data retained so that the data can be restored after loss, corruption, or other failure.
• concurrencyConcurrency is the handling of multiple operations or requests that execute or overlap during the same period.
• Google Cloud StorageGoogle Cloud Storage is Google's fully managed object-storage service for storing and accessing data as objects.
• complianceA compliance posture is an organization's current condition of adherence to applicable laws, regulations, policies, and data-handling requirements.
• identity providerAn identity provider is a system that authenticates users or services and supplies their identities and authorization context to other systems.
• Kafka ConnectKafka Connect is a framework for continuously transferring data between Apache Kafka and external systems through connectors.
• total cost of ownershipTotal cost of ownership is the complete cost of acquiring, operating, maintaining, and retiring a system or asset.
• catalog-level branchingCatalog-level branching is the creation of independent metadata histories or views for catalog-managed tables without copying their underlying data.
• observability toolAn observability tool collects and presents operational signals that reveal a system's internal state and behavior.
• ingestion layerAn ingestion layer is a system layer that collects data from source systems, transforms and cleans it, and loads it into storage tables.
• vendor lock-inTool lock-in is dependence on a particular software tool or vendor that makes changing tools difficult or costly.
• storage classA storage access tier is a storage classification that associates data with an access pattern and corresponding cost and performance characteristics.
• Microsoft Entra IDAzure Active Directory is Microsoft's cloud identity and access management service for authenticating identities and enforcing authorization policies.
• dead letter queueA dead-letter queue is a side channel that stores records rejected or unprocessable by a normal data-processing path for later inspection or correction.
• alertingAlerting is the automated generation and delivery of notifications when monitored data satisfies configured conditions.
• WasabiWasabi is a managed cloud object-storage service designed for storing and accessing data through an S3-compatible interface.
• dbtdbt is an open-source tool for transforming data in analytical data warehouses and lakehouses using SQL-based models, testing, and documentation.
• deletion semanticsDeletion semantics are the rules determining how a deletion changes logical table state and when it removes corresponding physical data.
• Arrow Database ConnectivityADBC is an Arrow-based database connectivity interface for high-performance access to analytical data systems.
• reliabilityReliability is the ability of a system to perform consistently and preserve correct results despite failures, retries, and interruptions.
• technology auditA technology audit is a systematic examination of an organization's technology systems, configurations, constraints, and alignment with organizational needs.
• data lossData loss is the destruction or unavailability of data that prevents its intended recovery or use.
• scalabilityScalability is the ability of a system to handle increasing data volume or demand while maintaining acceptable operation through manageable expansion.
• snapshot historySnapshot history is the ordered record of prior versioned states maintained for a table.
• identity and access managementIdentity and access management is the discipline and technology for authenticating identities and controlling their access to resources.
• Microsoft OneLakeMicrosoft OneLake is the unified logical data-lake storage layer underlying the Microsoft Fabric analytics platform.
• HiveApache Hive is a data warehouse infrastructure that provides SQL-based querying and table management over distributed data stored in file systems or object stores.
• metadata managementMetadata management is the discipline of collecting, organizing, governing, and maintaining descriptive information about data assets.
• snapshot retention policyA snapshot retention policy defines which table snapshots remain available and the conditions under which older snapshots are expired.
• Microsoft FabricMicrosoft Fabric is an integrated cloud analytics platform providing data integration, engineering, warehousing, and business-intelligence capabilities.
• flat namespaceA flat namespace addresses objects directly by unique keys without organizing them in nested directories.
• controlled castingControlled casting is the deliberate coercion of incoming values into types compatible with a target schema under defined rules.
• fail-fastFail-fast ingestion rejects records or halts processing when incoming data does not conform to the expected schema.
• schema alignmentSchema alignment is the process of reconciling incoming fields and types with the definition of a target table.
• variant data typeA variant data type represents semistructured values whose fields or shapes may vary across records.
• Lightweight Directory Access ProtocolThe Lightweight Directory Access Protocol is a network protocol for accessing and managing directory information, including user identities.
• data qualityData quality is the degree to which data is accurate, consistent, complete, timely, and fit for its intended use.
• deduplicationDeduplication is the removal or suppression of repeated records or events representing the same logical occurrence.
• upsertAn upsert operation inserts records that are absent and updates records that already exist according to a matching key or condition.
• ETLA workflow that extracts data from sources, transforms it, and loads it into a target data system.
• catalog backupThe practice of exporting, synchronizing, or otherwise backing up catalog definitions and metadata references for later restoration.
• snapshot isolationSnapshot isolation is a concurrency-control technique in which a transaction reads from a consistent version of data while concurrent changes are isolated from that view.
• strong consistencyStrong consistency is a storage property in which successful writes become immediately visible to subsequent reads and operations observe a single current state.
• Hadoop catalogA catalog implementation that resolves Apache Iceberg tables from filesystem paths and stores their metadata without requiring a separate catalog service.
• attribute-based access controlAttribute-based access control is an authorization technique that makes permission decisions from attributes of users, resources, actions, or context.
• snapshot managementSnapshot lifecycle management governs the creation, retention, expiration, and removal of versioned table states.
• metadata governanceMetadata governance is the policies, controls, and practices for ensuring that metadata is accurate, secure, traceable, and consistently managed.
• catalog replicationCatalog replication is the export, synchronization, or copying of catalog state between repositories or environments.
• data sovereigntyData sovereignty is the principle that data is subject to the laws and governance requirements of the jurisdiction where it is stored or processed.
• metadata rewriteMetadata rewrite is the process of changing table metadata file paths so they reference replicated storage locations.
• multi-environment recoveryMulti-environment recovery is the restoration of data and system operation across isolated development, staging, production, or disaster-recovery environments.
• split-brainA split-brain scenario is a distributed-system condition in which multiple systems independently believe they own the authoritative state of the same resource.
• commit coordinationCommit coordination is the process of ordering and validating changes from writers so that metadata updates are applied safely and consistently.
• catalog requirementsCatalog requirements are the functional and nonfunctional capabilities specified for selecting, deploying, governing, and operating a lakehouse catalog.
• change-data-captureChange data capture processing handles records representing inserts, updates, and deletes made to a source system.
• external catalogAn external catalog is a catalog synchronized from another catalog implementation and exposed as read-only to non-Snowflake engines.
• internal catalogAn internal catalog is a catalog whose table registrations and metadata operations are managed within the catalog service.
• Apache GravitinoAn open-source catalog and metadata-management system designed to provide unified access to distributed data assets.
• multi-engine architectureA multi-engine architecture runs workloads on separate configurable query-execution clusters.
• QlikQlik Sense is a business-intelligence and analytics platform for interactive data exploration, visualization, and reporting.
• S3FileIOS3FileIO is an Apache Iceberg file I/O implementation for reading and writing table data through S3-compatible object storage.
• Azure Blob StorageAzure Blob Storage is Microsoft's managed object-storage service for storing and accessing large amounts of unstructured data.
• branching and taggingBranching and tagging is a version-reference technique that isolates mutable data changes and names stable data states.
• hierarchical namespaceA hierarchical namespace organizes object-storage paths into directory-like structures and enables filesystem-style operations.
• snapshot referenceA snapshot reference is a named pointer to a particular table snapshot.
• snapshot taggingSnapshot tagging is the assignment of persistent labels to selected table snapshots so that important historical states can be identified and protected.
• VACUUM TABLEVACUUM TABLE is a Dremio SQL command that expires old Apache Iceberg snapshots and removes their unreferenced files.
• PySparkPySpark is the Python API for Apache Spark distributed data processing.
• semantic consistencySemantic consistency is the preservation of the same meanings, entities, measures, and business definitions across data-access tools and interfaces.
• optimistic concurrency controlA concurrency-control approach that allows operations to proceed without locks and detects conflicts when changes are committed.
• workflow transparencyThe property of a workflow being understandable through explicit steps, documented inputs and outputs, and limited hidden behavior.
• RedpandaRedpanda is a Kafka-compatible distributed event-streaming platform designed for low-overhead, high-performance message processing.
• merge operationA merge operation conditionally inserts, updates, or deletes target records based on matches with a source dataset.
• columnar dataA columnar format stores values from the same field together to support efficient analytical reading and compression.
• state managementState management is the persistence, tracking, and restoration of intermediate processing state in a long-running data workflow.
• cloud-managed deploymentA cloud-managed deployment is an arrangement in which a provider operates the software while the customer uses it through managed cloud infrastructure and services.
• inline compressionInline compression is the automatic compression of data as it is written to storage, with decompression when it is retrieved.
• server-side encryptionServer-side encryption is encryption performed by a storage service before data is persisted and decrypted when authorized data is accessed.
• append modeAppend mode is a write configuration that adds new records to existing table data without replacing prior records.
• overwriteA table write operation that replaces existing data in a specified scope with new data.
• HIPAAHIPAA is a United States law establishing requirements for protecting the privacy and security of individually identifiable health information.
• zero-copyZero-copy data exchange transfers data between compatible systems without copying the underlying in-memory representation.
• hierarchical execution modelA query-execution technique that organizes processing into hierarchical stages to coordinate work across distributed data sources.
• small fileA small file is a data file whose size is sufficiently low that file-management, metadata, and scheduling overhead become significant relative to its data content.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.