Encryption converts readable data into a form that requires a key to decode. In Kafka, deciding what to encrypt is a deployment decision: client-to-broker traffic, internal cluster traffic, and data stored on broker disks face different risks. Encryption complements authentication and authorization; it does not decide who may access a topic. Kafka is not secured by default, so the chosen protections must be configured. section 9.3.1 section 9.2.3
Choose protection for each point in the data’s path:
- In transit: TLS protects communication between clients and brokers. Assess broker-to-broker and broker-to-controller traffic separately rather than assuming every connection has the same exposure. section 9.0 section 9.2.3
- At rest: transport encryption does not encrypt Kafka’s stored messages. A broker decrypts incoming data, stores it in plaintext, and re-encrypts it for delivery. Disk or filesystem encryption, such as LUKS, or a cloud-managed service can protect stored data. section 9.3.5
- End to end: producers can encrypt message contents before sending them, leaving consumers to decrypt after retrieval. This requires a custom implementation and a key management service for key creation, issuance, and revocation. Keep partition keys unencrypted to preserve distribution when encryption keys change; message headers can carry encryption parameters. Every participant must support the policy, which may be difficult with third-party software. section 9.3.5
Customer 360 deployment
A Customer 360 operational data store uses mTLS for exposed client-to-broker connections, accepting certificate-management work in return for authentication and encryption. Inside a tightly segmented and monitored private network, it uses SASL_PLAINTEXT for broker and controller traffic to avoid TLS overhead. That is an explicit acceptance of internal traffic risk, not a general rule that internal encryption is unnecessary. The infrastructure team issues keys and certificates and documents the network connections. section 9.3.6
Table 9.2 Communication for Customer 360 components
| Client |
Server |
Port |
Security protocol |
Access type |
|---|---|---|---|---|
| Producer |
Broker |
9093 |
mTLS |
Write to topics |
| Consumer |
Broker |
9093 |
mTLS |
Read from topics |
| Broker |
Broker |
9092 |
SASL_PLAINTEXT |
Replication |
| Controller |
Controller |
9094 |
SASL_PLAINTEXT |
Exchange metadata for cluster management |
| Broker |
Controller |
9094 |
SASL_PLAINTEXT |
Send metadata updates and leadership election |
| Kafka Connect |
Broker |
9093 |
mTLS |
Read/write data to/from connector topics |
| Schema Registry |
Broker |
9093 |
mTLS |
Read/write data to/from |
| Producer |
Schema Registry |
8081 |
HTTPS |
Retrieve schema for serialization |
| Consumer |
Schema Registry |
8081 |
HTTPS |
Retrieve schema for deserialization |
| Kafka Connect |
Schema Registry |
8081 |
HTTPS |
Retrieve schema for serialization and deserialization |
Compare the Customer 360 connections by client, server, port, and security protocol to see where mTLS and SASL_PLAINTEXT are applied. section 9.3.6
Set the policy before configuring protocols: determine whether organizational rules require encryption in transit, at rest, or for sensitive fields within a payload. Weigh compliance and exposure against performance and operational complexity, and involve security and network teams in the decision. section 10.1.4 section 9.2.3
