End-to-end encryption

As explained in Kafka for Architects

End-to-end encryption is a communication design in which data is encrypted by the producer or sender and decrypted only by the intended consumer or recipient.

End-to-end encryption (E2EE) protects message contents across a Kafka pipeline by having producers encrypt before publication and consumers decrypt after retrieval. Unlike client–broker encryption, the broker never sees plaintext, so encrypted data can remain protected on broker disks. section 9.3.5

A Kafka E2EE design carries several architectural and operational consequences:

• Kafka provides no built-in E2EE mechanism; custom application logic, client-library integration, or an intermediary proxy is required. section 9.3.5
• Use a key management service for key creation, issuance, and revocation, and ensure every participating component supports the encryption policy. section 9.3.5
• Avoid encrypting partition keys when stable data distribution matters; place encryption parameters in message headers rather than the payload when appropriate. section 9.3.5
• Treat E2EE as complementary to authentication, authorization, and TLS rather than as a replacement for them. section 9.0 section 9.5

Protected Kafka message flow

A producer encrypts a regulated message before sending it to Kafka. The broker stores and forwards the ciphertext without processing its contents, and an authorized consumer decrypts it after retrieval. A KMS supplies and revokes the relevant keys; a proxy can centralize encryption and decryption when client changes are difficult. section 9.3.5

Figure 9.12 Implementing end-to-end encryption with the encryption and decryption of messages done on the clients

Client-side encryption and decryption across the Kafka message path. section 9.3.5

How Kafka for Architects covers it

How deep it goes

Explains producer–consumer flow, partition-key constraints, KMS needs, compatibility issues, and proxy alternatives without implementation code.

Examples

Uses client-side encryption diagrams and contrasts filesystem, cloud-storage, application, library, and proxy-based protection.

What it assumes

Assumes Kafka producers, consumers, brokers, partitions, topics, and client–broker security concepts.

What you can do after

Operate/configure a Kafka E2EE design by selecting client or proxy handling, preserving partitioning, and integrating a KMS.

How it connects

Connects E2EE with Kafka brokers, producers, consumers, partitioning, message headers, key management, proxies, TLS, authentication, and authorization.

liveBookmanning.com

Read Kafka for Architects on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Real-World CryptographyEnd-to-end encryption is a communication security technique in which only the communicating endpoints can decrypt the exchanged content.
• 

  liveBookmanning.com
  Apache Kafka in ActionEnd-to-end encryption is encryption in which data is encrypted by the originating endpoint and decrypted only by the intended receiving endpoint, without readable intermediaries.

Compare how 3 books explain end-to-end encryption →

Related concepts

• cryptographic protocolcontrasts with · A cryptographic protocol is a defined sequence of cryptographic operations and message exchanges that provides security properties between communicating parties.
• platformused by · A platform is a foundational computing system, environment, or online service that provides capabilities or infrastructure for building, running, delivering, or interacting with applications and services.
• public-private key pairuses · A public-private key pair is a mathematically related public key and private key used together for asymmetric cryptographic operations.
• TLScontrasts with · Transport Layer Security (TLS) is a cryptographic protocol that protects network communications through confidentiality, integrity, and authentication of communicating endpoints.
• encryptionkind of · Encryption is the cryptographic transformation of readable data into ciphertext to conceal its contents, reversible through decryption with appropriate information.
• browseruses · A web browser is software that retrieves, interprets, and renders web resources, executes supported web content, and enables interaction with web pages and applications.

This explainer was generated by Manning's knowledge graph from the text of Kafka for Architects. The section links point to the parts of the book it draws on.