liveBookmanning.com

What you will learn in Apache Kafka in Action

From basics to production

by Anatoly Zelenin, Alexander Kropp
Foreword by Adam Bellemare

Read it on liveBook Get the book at manning.com

Chapter by chapter

Concepts explained in this book

• Apache KafkaApache Kafka is a distributed event-streaming platform for publishing, storing, processing, and subscribing to streams of records.
• partitionA Kafka partition is an ordered, append-only subdivision of a topic that stores records and enables parallel processing.
• Kafka brokerA Kafka broker is a server process that stores and serves Kafka records and participates in cluster operations.
• monitoringA log is an append-oriented sequence of records that captures occurrences or changes over time.
• topicA Kafka topic is a named stream of records to which producers publish and from which consumers read.
• clientA Kafka client is an application that connects to a Kafka cluster to publish, consume, or otherwise interact with Kafka data.
• Kafka ConnectKafka Connect is a framework for reliably moving data between Kafka and external systems through connectors.
• Kafka producerA producer is an application or service that publishes records to Kafka topics.
• streamingStream processing is the continuous computation of results over records as they arrive.
• batchingA batch is a group of Kafka records collected into one unit for transmission or storage.
• Kafka clusterA Kafka cluster is a group of Kafka broker nodes that jointly store and serve event data, with security controls determining how clients and brokers connect and access it.
• in-sync replicaAn in-sync replica is a Kafka partition replica whose data is up to date with the leader.
• joinA join is a query operation that combines related records from multiple data sources according to matching or related fields.
• offsetAn offset is a position marker identifying a record's location within a Kafka partition or a connector's processed source sequence.
• partitioning keyA message key is an optional Kafka message attribute used to categorize a message and determine its partition.
• consumer groupA consumer group is a set of consumers that collaboratively consume topic partitions, with each partition assigned to one consumer in the group.
• primary nodeA Kafka partition leader is the broker responsible for handling reads and writes for a partition.
• partition countA partition count is the configured number of partitions belonging to a Kafka topic.
• replicationReplication is the maintenance of multiple copies of data across system components for availability and resilience, with asynchronous replication allowing temporary divergence between copies.
• recordA Kafka record, also called a message, is a data item published to and consumed from a Kafka topic.
• dataset splitSharding divides data across multiple subsystems or database nodes, typically by a shard key.
• throughputThroughput is the amount of data or number of messages processed or transferred per unit of time.
• consumer group rebalancingA rebalance is the redistribution of consumer-group partition assignments after membership, partition, or consumer availability changes.
• Kafka StreamsKafka Streams is a Java library for building applications that process Kafka records as streams.
• ZooKeeperA ZooKeeper ensemble is a group of ZooKeeper nodes that collectively provide distributed coordination and metadata services.
• replication factorA replication factor is the number of copies maintained for each Kafka partition.
• all acknowledgmentacks=all, also represented as acks=-1, is a Kafka producer setting that waits for all in-sync replicas to acknowledge a message.
• KRaftKRaft is Kafka's protocol for managing cluster metadata and internal coordination without an external coordination system.
• databaseA database is a software system that durably stores, organizes, and supports querying of data.
• log segmentA log segment is a file containing a contiguous portion of the records in a Kafka partition log.
• Order propertyMessage order is the sequence relationship in which messages are stored or processed.
• controllerA Kafka controller is a broker designated to manage partition and replica state, including partition assignments and leader selection.
• time windowA time window is a bounded temporal interval over which stream records are aggregated, correlated, or otherwise processed.
• asynchronous programmingAsynchronous data exchange transfers information without requiring communicating components to remain synchronized or wait for one another to complete.
• log compactionLog compaction is a Kafka cleanup technique that retains the latest record for each message key and removes obsolete records.
• backward compatibilityBackward compatibility is schema compatibility in which consumers using the current schema can read data written with older schemas.
• replicaA Kafka replica is a copy of a partition maintained on a broker for fault tolerance and high availability.
• Kafka transactionA Kafka transaction is an atomic operation that groups message writes and related offset commits so they either all succeed or all roll back.
• round-robin load balancingRound-robin distribution assigns successive records or work items to available partitions or recipients in cyclic order.
• reliabilityReliability is the likelihood that a system continues to perform correctly over a specified period and operating conditions.
• kafka-topics.shkafka-topics.sh is an Apache Kafka command-line utility for creating, altering, describing, and deleting topics.
• consumer lagA consumer lag metric reports replication lag separately for each follower, topic, and partition.
• schemaA schema is a formal blueprint that defines the structure, fields, types, and rules of data.
• authorizationAccess control is the enforcement of rules that determine which principals may access particular data or perform particular operations.
• parallelismParallel processing is the simultaneous execution of work across multiple processing units or workers.
• change-data-captureChange data capture is a technique that monitors and records changes made to a data source rather than repeatedly reading its complete current contents.
• follower nodeA Kafka partition follower is a broker that copies partition data from the leader for resilience and failover.
• real-time processingReal-time data processing handles data with sufficiently low latency for systems to react soon after events occur.
• data aggregationA metric aggregation combines metric observations over a time period or other grouping to produce a summarized value.
• connectorA Kafka Connect connector is a component that transfers data between Kafka and an external system according to a configured connection definition.
• __consumer_offsets__consumer_offsets is Kafka's internal topic for storing consumers' current reading positions.
• RESTREST is an architectural style for networked applications that represents resources and accesses them through stateless, uniform interfaces.
• alertingAn alerting mechanism is a system function that detects specified conditions and notifies operators when they occur.
• acknowledgmentAn acknowledgment (ack) is a Kafka broker response indicating the configured level of message receipt or persistence.
• min.insync.replicasThe minimum in-sync replicas setting specifies the smallest number of replicas that must acknowledge a write for it to succeed.
• DebeziumDebezium is an open-source distributed platform that captures row-level changes from databases and emits them as event streams.
• stateCurrent state is the complete information representing the present condition of a system or entity.
• consumer assignment strategyA partition assignment strategy is a consumer setting that selects the algorithm used to assign topic partitions among members of a consumer group.
• CooperativeStickyAssignorThe Cooperative Sticky Assignor is a Kafka partition-assignment algorithm that minimizes assignment changes and permits unaffected partitions to keep being consumed during rebalancing.
• resource limitA Kafka quota is a resource limit applied to clients or users to control request rates or broker resource consumption.
• schema governanceSchema management is the practice of defining, documenting, governing, and maintaining schemas for data formats.
• group.idgroup.id is the Kafka consumer configuration property that identifies the consumer group to which a consumer belongs.
• batch sizeA batch size is the maximum amount of record data grouped into a producer batch before transmission.
• messaging systemA messaging system is software that transfers messages between producers and consumers, typically providing delivery and integration capabilities.
• data meshA data mesh is a decentralized data architecture in which domain-oriented teams own and provide data as products through shared self-service infrastructure and federated governance.
• leader electionLeader election is the process of selecting a node to act as leader among eligible nodes in a distributed system.
• authenticationAuthentication is the process of verifying the identity of a user or system requesting access to a resource.
• group leaderA group leader is the first consumer to join a consumer group and the member responsible for creating its partition distribution plan.
• repartitionRepartitioning redistributes stream records according to their keys so that records requiring keyed processing reach the appropriate task.
• exactly-once deliveryExactly-once semantics guarantee that each input message produces one durable effect in the destination despite retries or failures.
• data formatA data format is a specification for representing and organizing data for storage, transmission, and interpretation.
• acks=0acks=0 is a Kafka producer setting that sends no broker acknowledgment to the producer and provides no guarantee that the message is persisted.
• changelogA changelog topic is an internal Kafka topic that durably records updates to processing state so the state can be restored after a failure.
• KStreamA KStream is a Kafka Streams abstraction representing an unbounded stream of records, where each record is treated as an independent event.
• acknowledgment strategyAn acknowledgment strategy is a Kafka producer policy or setting that determines when a broker acknowledges a published message.
• log retentionLog retention is a Kafka storage policy that removes records according to retention criteria such as age or size rather than record keys.
• idempotencyAn idempotent write operation can be retried or applied repeatedly without producing effects beyond those of a single application.
• stream processorA stream processor is a component that examines, transforms, combines, produces, or routes records in a continuously processed data stream.
• fault toleranceFault tolerance is the ability of a system to continue operating correctly despite failures in components or infrastructure.
• synchronous callSynchronous communication is an interaction in which the sender and receiver coordinate during the same exchange and the sender waits for a response.
• timestampA timestamp is a recorded indication of when a Kafka message or record was produced or otherwise associated with a time.
• distributed commit logA persistent distributed log is an append-oriented record of ordered events replicated across multiple machines and retained for later reading.
• data productA data product is a managed, discoverable data asset provided by a domain team with defined quality, access, and interface expectations.
• metadataMetadata is descriptive information about Kafka topics, brokers, partitions, replicas, assignments, and leadership.
• producer idempotenceenable.idempotence is a Kafka producer setting that enables duplicate-safe retries of records.
• master dataMaster data is relatively stable, authoritative information describing core business entities such as customers or products.
• coordination clusterA coordination cluster is a group of nodes that monitors a distributed cluster and manages its membership, metadata, assignments, and coordination state.
• publish/subscribe modelA publish-subscribe model lets publishers send messages to named channels while subscribers independently receive messages from those channels.
• state storeA Kafka Streams state store is a local storage component that maintains processing state for a stream-processing application.
• repartition topicA repartition topic is an internal Kafka topic created to temporarily redistribute records by key before downstream keyed processing.
• linger.mslinger.ms is a Kafka producer setting that specifies how long a producer waits for additional records before sending a batch.
• countcount is a Kafka Streams aggregation operation that maintains the number of records for each key or grouping and produces aggregate results.
• high-water markA high water mark is the highest offset in a Kafka partition whose record is available for consumption because the required replicas have received it.
• transaction logA transaction log is an append-oriented database record of transactions and the data changes they produce.
• unclean leader electionUnclean leader election is the promotion of a non-in-sync replica to partition leader when no eligible in-sync replica is available.
• key-based distribution strategyKey distribution is the pattern by which message keys and their associated records are spread across partitions.
• worker configurationA worker configuration is a set of settings that controls Kafka Connect worker resources, task distribution, and integration behavior.
• Flink SQLFlink SQL is Apache Flink's SQL interface for defining continuously evaluated queries over streaming and table-oriented data.
• taskA Kafka Streams task is the smallest processing unit, responsible for processing one or more assigned partitions and executing a topology.
• connector configurationConnector configuration is the set of named parameters that defines how a Kafka Connect connector connects to and processes an external system.
• stretched clusterA stretched cluster is a distributed cluster whose nodes are deployed across separate physical locations or failure domains.
• JDBC source pluginA JDBC source connector is a Kafka Connect connector that reads records from relational databases through JDBC and publishes them to Kafka.
• stream-table joinA stream-table join combines each incoming stream record with the table value associated with the record's key.
• table-to-table joinA table-table join combines two changing tables and updates results when either table changes.
• Kafka Connect converterJsonConverter is a Kafka Connect converter that serializes and deserializes record data as JSON, optionally including schemas.
• retention policyA retention period is the duration for which Kafka retains records before allowing them to be deleted.
• schema registryA schema registry is a service that stores, versions, and provides access to schemas governing data formats.
• immutable collectionImmutability is the property of data that prevents its existing contents from being changed or removed after creation.
• Apache MirrorMakerKafka MirrorMaker 2 is Kafka's tool for replicating topics, consumer offsets, and related metadata between Kafka clusters.
• securitySecurity is the protection of systems, data, and services against unauthorized access, misuse, disruption, or compromise.
• consistencyData consistency is the property that maintained copies or views of data agree with the intended current data.
• encryptionEncryption transforms readable data into an encoded form that requires a corresponding decryption method to recover.
• ordering guaranteeAn ordering guarantee is a promise that records are observed in a specified sequence, such as their order within a partition.
• single message transformationA single message transformation is a Kafka Connect transformation that modifies an individual record as it passes through a source or sink connector.
• access control listAn access control list is a collection of rules specifying which identities may perform operations on protected resources.
• throttle-time metricA throttle-time metric reports the time a Kafka broker delays a client because the client has exceeded a quota.
• kafka-console-producerkafka-console-producer.sh is an Apache Kafka command-line utility that publishes records to a topic from console input.
• fetch.min.bytesfetch.min.bytes is a Kafka consumer setting that specifies the minimum number of bytes the broker should accumulate before responding to a fetch request.
• mapValuesmapValues is a stream-processing function that transforms record values while preserving their keys.
• managed Kafka serviceA managed Kafka service is a hosted offering in which a provider operates Kafka infrastructure for customers.
• eventAn event is a record representing a significant occurrence, change, activity, transaction, or incident and its associated information.
• data lossData loss is the unintended destruction or unavailability of stored data.
• page cacheA page cache is an operating-system memory cache that stores file data in memory to accelerate reads and defer writes to storage.
• filteringfilter is a stream-processing function that retains only records for which a predicate returns true.
• mergemerge is a stream-processing function that combines records from multiple streams into one stream.
• mapmap is a stream-processing function that transforms each input record into a new record or value.
• splitsplit is a stream-processing function that routes records into multiple branches according to predicates or conditions.
• log end offsetA log end offset is the position immediately after the last message received in a Kafka partition log or replica log.
• backup and restoreA backup is a separately retained copy of data that can be used to restore the original after loss or corruption.
• segment roll interval settinglog.roll.ms is a Kafka broker configuration that specifies the default time interval for rolling log segments.
• log cleanerThe Kafka log cleaner is a Kafka broker subsystem that compacts or removes records from partition logs according to configured cleanup rules.
• streaming queryStreaming SQL is SQL applied continuously to arriving data streams, producing results as data changes.
• plugin.pathplugin.path is a Kafka Connect setting that specifies filesystem paths where connector, converter, and transformation plugins are located.
• sink connectorA sink connector is a Kafka Connect connector that reads data from Kafka and writes it to an external destination system.
• source connectorA source connector is a Kafka Connect connector that reads data from an external source system and publishes it to Kafka.
• schema validationSchema validation is the process of checking data against the structure and rules defined by a schema.
• CPU timeCPU time is the amount of processor execution time consumed by a client's requests on a broker.
• RangeAssignorThe Range Assignor is a Kafka partition-assignment algorithm that assigns corresponding partition numbers from subscribed topics to the same consumers.
• RoundRobinAssignorThe Round Robin Assignor is a Kafka partition-assignment algorithm that distributes partitions from subscribed topics evenly among consumers.
• leader-follower replication modelThe leader-follower principle assigns one replica as the leader that handles coordination and other replicas as followers that copy its data.
• leader rebalancingLeader rebalancing is the redistribution of Kafka partition leadership to restore preferred leader assignments or balance broker load.
• preferred leader electionPreferred leader election is the selection of a partition's designated preferred replica as its leader when that replica is available and synchronized.
• i/o threadAn I/O thread is a Kafka broker thread that performs broker input and output processing.
• data enrichmentData enrichment is the process of augmenting records with related attributes from another data source.
• lookup operationA lookup is the retrieval of a corresponding value from stored state by a key or identifier.
• isolationisolation.level is a Kafka consumer configuration that determines whether records from uncommitted transactions can be read.
• under-minimum-isr partition count metricAn under-minimum ISR is a metric indicating that a partition's in-sync replica set is below its configured minimum size.
• Kafka Connect clusterA Kafka Connect cluster is a group of Kafka Connect instances with the same group identifier that coordinate and distribute connector load.
• retryA retry is a technique or mechanism that repeats an operation after a failure or timeout in an attempt to complete it successfully.
• bandwidth quotaA bandwidth quota is a Kafka quota that limits the rate at which a client can produce or consume data.
• network throttlingThrottling is the deliberate delaying or muting of client requests to keep resource usage within an enforced limit.
• fetch requestA fetch request is a Kafka protocol request for retrieving records from a partition.
• consensusConsensus is distributed agreement among nodes on a shared value or system state.
• streaming tableA stream processing table is a continuously updated key-value representation of current state derived from records in a stream.
• metadata log__cluster_metadata is Kafka's internal topic for storing cluster metadata across brokers in KRaft mode.
• RaftRaft is a distributed consensus algorithm and protocol for coordinating replicated state machines so cluster nodes agree on replicated state.
• subtopologyA subtopology is a portion of a Kafka Streams processing topology separated to write to or read from an intermediate topic.
• kafka-console-consumerkafka-console-consumer.sh is an Apache Kafka command-line utility that reads records from a topic and displays them on the console.
• broker.rack propertybroker.rack is a Kafka broker configuration that assigns a rack or location identifier to a broker.
• client.rackclient.rack is a Kafka client configuration that assigns a rack or location identifier used to select a nearby replica for fetching records.
• KIP-392KIP-392 is a Kafka Improvement Proposal that enables consumers to fetch records from the replica nearest to the client.
• migrationA ZooKeeper-to-KRaft migration is the controlled transition of Kafka cluster metadata management from ZooKeeper to KRaft.
• partitionerA partitioner is the Kafka component that selects the partition to which a record is sent.
• Kafka Connect workerA Kafka Connect worker is a process in a Kafka Connect cluster that manages connectors and runs their tasks.
• network threadA network thread is a broker thread that handles network communication and contributes to the broker's request-processing capacity.
• principalA principal is the authenticated security identity representing a user or system when accessing protected resources.
• stateless systemA stateless processor computes each output from the current input record without retaining processing state between records.
• partition directoryA partition directory is a broker filesystem directory containing the files for a Kafka partition assigned to that broker.
• under-replicated partitionAn under-replicated partition is a Kafka partition that currently has fewer replicas than its configured replication requirement.
• transaction coordinatorA transaction coordinator is the Kafka broker component that coordinates a transaction and writes markers recording its commit or abort outcome.
• KTableKTable is the Kafka Streams abstraction representing the latest value for each key as a continuously updated table.
• data streamA data stream is a continuous or unbounded sequence of data records processed as they become available.
• client metricA client metric is an operational measurement describing the behavior, health, resource use, or traffic of a Kafka client.
• SQLAn SQL query is a declarative statement that specifies data to retrieve, transform, or aggregate.
• StrimziThe Strimzi Cluster Operator is a Kubernetes controller that deploys and manages Kafka clusters and their associated components.
• partition reassignmentPartition reassignment is the Kafka operation of changing the brokers assigned to store topic partitions and their replicas.
• partition reassignment toolkafka-reassign-partitions.sh is an Apache Kafka command-line utility for reassigning partitions and their replicas, including across multiple topics.
• at-least-once deliveryAt-least-once delivery is a message delivery guarantee that prevents message loss but permits duplicate delivery.
• at-most-once deliveryAt-most-once delivery is a message delivery guarantee that permits message loss but prohibits duplicate delivery.
• Kafka-specific metricA Kafka Streams metric is a measurement describing the performance or operation of a Kafka Streams application.
• event-driven architectureEvent-driven architecture is a software architecture in which components communicate by producing and consuming representations of events.
• metadata APIA metadata request is a Kafka client request that retrieves information about topics, partitions, brokers, and partition leaders.
• JDBC connectorA JDBC sink connector is a Kafka Connect connector that writes records from Kafka to a relational database through JDBC.
• decentralized governanceDecentralized governance is a governance model in which data standards and controls are established collaboratively across domains rather than imposed solely by a central team.
• domain ownershipDomain ownership is the assignment of responsibility for producing, maintaining, documenting, and governing data to the business domain that understands it.
• self-service data infrastructureSelf-service data infrastructure is shared technical infrastructure that lets domain teams independently publish, consume, modify, and operate data capabilities without case-by-case intervention from a central data team.
• alert fatigueAlert fatigue is reduced attention or responsiveness caused by receiving excessive, repetitive, or low-value alerts.
• tasks.maxtasks.max is a Kafka Connect configuration property that sets the maximum number of tasks into which a connector workload may be divided.
• group coordinatorA group coordinator is the Kafka broker responsible for managing a specific consumer group's membership, liveness, and assignments.
• forward compatibilityForward compatibility is schema compatibility in which consumers using an older schema can read data written with a newer schema.
• none compatibilityNo compatibility is a schema policy in which consumers must remain synchronized with the producer and cannot rely on reading data across schema versions.
• StickyAssignorThe Sticky Assignor is a Kafka partition-assignment algorithm that preserves as many existing partition assignments as possible during rebalancing.
• migration modeMigration mode is a Kafka configuration state that enables the transition of cluster metadata management from ZooKeeper to KRaft.
• schema versionA schema version is a distinct revision of a data schema maintained as schemas change over time.
• scalabilityScalability is the ability of a system to maintain or improve its service as workload or resource capacity increases.
• sequence numberA sequence ID is an ordered identifier assigned to a Kafka message to detect duplicates and verify message order within a partition.
• Flink Kafka connectorA Flink Kafka connector is a connector component that enables Apache Flink to read data from or write data to Kafka.
• mutual TLSMutual TLS is a TLS configuration in which both communicating endpoints authenticate each other with certificates.
• data on the insideData on the inside is data managed within a service and structured to fit that service's specific needs.
• external inputData on the outside is service-external data structured for exchange across service boundaries and independent of any one service's internal model.
• schema evolutionSchema versioning is the management of successive schema definitions as data structures evolve over time.
• governanceGovernance is the structured oversight of data, security, and resource practices that controls how a technical system is operated and maintained.
• leader epochA leader epoch is a monotonically increasing Kafka partition leadership version that distinguishes current leadership from stale leadership.
• leader epoch checkpoint fileA leader epoch checkpoint file is a Kafka partition-directory file that records leader-epoch information and the log offset associated with leader selections.
• memory boundA bottleneck is a component or resource that limits the throughput or capacity of a system.
• Cruise ControlCruise Control is a Kafka ecosystem tool for monitoring cluster load and automating partition reassignment and resource balancing.
• load balancingLoad balancing is the distribution of work or resource consumption across system components to avoid substantial imbalance.
• broker failureA broker failure is a failure or malfunction affecting a Kafka broker and its ability to serve partition data.
• topic alterationTopic alteration is the Kafka operation that changes supported properties of an existing topic.
• topic creationTopic creation is the Kafka operation that creates a named topic with configured partitions and replicas.
• topic deletionTopic deletion is the Kafka operation that removes a topic and its stored records.
• data tieringTiered storage is a storage strategy that places data on different storage tiers according to factors such as age, cost, or access frequency.
• Kafka client libraryA Kafka client library is a language-specific software library that lets applications communicate with Kafka brokers.
• majorityA majority is more than half of the members of a group, sufficient for quorum-based agreement in a distributed system.
• commandA command is a message that requests another system to perform a specific action.
• deltaA delta is a message representation containing only the changes to an object rather than its complete state.
• AvroAvro is a schema-based data serialization system that encodes structured data in a compact binary format.
• stream processing systemA stream processing system is software that operates directly on continuously arriving data streams.
• distributed systemA distributed system is a set of computing components that coordinate across multiple machines to provide a unified service.
• share groupA share group is a Kafka consumption pattern in which multiple consumers can cooperatively consume records from the same partition.
• resource utilizationCPU utilization is the proportion of processor capacity occupied by executing work over a measurement interval.
• network bandwidthBandwidth is the rate at which data can be transmitted or processed by a system or communication channel.
• core systemA core system is an essential central business application that supports an organization's primary operational processes.
• end-to-end encryptionEnd-to-end encryption is encryption in which data is encrypted by the originating endpoint and decrypted only by the intended receiving endpoint, without readable intermediaries.
• durabilityData durability is the ability of stored data to remain preserved and recoverable despite failures or restarts.
• incrementing modeIncrementing mode is a JDBC source connector polling mode that detects new rows using a strictly increasing column value.
• timestamp modeTimestamp-based mode tracks database changes using a column containing modification timestamps.
• TLSTransport Layer Security (TLS) is a cryptographic protocol that provides authenticated, encrypted communication and protects network traffic from eavesdropping and tampering.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.