Kafka protocol

As explained in Kafka for Architects

The Kafka binary protocol is the binary request-and-response protocol used by Kafka clients and servers for communication.

Kafka’s binary protocol carries requests and responses between clients and brokers over TCP/IP. Client libraries hide serialization, transmission, acknowledgments, retries, and protocol-version negotiation while preserving common architectural principles across languages. section 3.1.1

The protocol supports an architecture in which:

• Producers serialize records, select partitions, assemble batches, send them to brokers, and optionally wait for acknowledgments before retrying failed or unacknowledged batches. section 3.1.1
• Clients and brokers negotiate the highest mutually supported API version, allowing independent upgrades when compatible versions exist. section 3.1.1
• High-level frameworks simplify integration, while low-level clients expose control over message conversion, dead-letter routing, concurrency, and offset handling. section 3.1

Customer 360 integration

A customer-profile service validates and publishes profile changes to Kafka, while another service consumes profile, billing, and interaction topics to build a consolidated profile. Both applications embed producer or consumer logic, commonly through a framework or client library. section 3.1

Protocol interoperability and browser limits

Azure Event Hubs accepts existing Kafka clients through the Kafka protocol, whereas browsers cannot connect directly because they lack raw TCP access; browser integrations therefore require an HTTP-based REST Proxy or another supported bridge. section 5.3.3 section 12.2.1

Figure 3.1 Simplified workflow of message transmission by a producer, including serialization, partition selection, batch assembly, and data transmission to the server, with an optional confirmation step

Trace the producer path from serialization and partition selection through batching, transmission, and optional broker confirmation. section 3.1.1

How Kafka for Architects covers it

How deep it goes

Explains transport, batching, acknowledgments, retries, and version negotiation, but stops short of byte-level request layouts.

Examples

Uses producer transmission, Customer 360 services, Azure Event Hubs interoperability, and browser connectivity constraints.

What it assumes

Assumes familiarity with producers, consumers, brokers, and messages while defining their protocol-mediated communication.

What you can do after

Operate/configure Kafka clients across languages and compatible broker versions while evaluating framework abstraction and direct-client trade-offs.

How it connects

Connects producers, consumers, brokers, TCP/IP, client libraries, REST Proxy, WebAssembly, and Azure Event Hubs.

liveBookmanning.com

Read Kafka for Architects on liveBook Get the book at manning.com

Related concepts

• Confluent REST Proxyused by · Confluent REST Proxy is a service that exposes Kafka data production, consumption, and administration through HTTP and REST interfaces.
• Apache Kafkaused by · Apache Kafka is a distributed event-streaming platform for publishing, storing, processing, and consuming streams of records.
• client libraryalternative to · A client library is reusable software that provides programmatic interfaces for an application to interact with a service, platform, or system.
• Kafka client libraryimplemented by · A Kafka client library is software that provides APIs and client-side behavior for applications to connect to Kafka and produce or consume records.
• network listenerconfigured by · A network listener is a configured component or endpoint that binds to a network address and protocol, accepts incoming traffic, and may forward it for processing.

This explainer was generated by Manning's knowledge graph from the text of Kafka for Architects. The section links point to the parts of the book it draws on.