End-to-end encryption protects highly sensitive Kafka data when transport encryption, authentication, authorization, and disk encryption do not protect against administrators with server or Kafka administrator rights. The producer encrypts the data, and only the intended consumer decrypts it; Kafka handles the message without readable contents. section 13.2 section 13.2.5
Figure 13.3 Securing a Kafka cluster involves transport encryption for communication, user authentication and authorization, and encryption of data at rest.
Use the security-layer diagram to distinguish end-to-end encryption from transport encryption, authentication, authorization, and encryption at rest. section 13.2
A practical Kafka implementation follows these principles:
- Implement encryption in producers and consumers because Kafka has no native end-to-end encryption support. section 13.2.5
- Wrap the regular serializer: serialize the message first, then encrypt the resulting byte array; reverse the order in the deserializer. section 13.2.5
- Manage encryption keys through a dedicated process, potentially using Vault. section 13.2.5 section 13.4
- Treat proxy-based encryption cautiously: a proxy can read messages when producer-to-proxy and proxy-to-consumer paths remain unencrypted. section 13.2.5
- Expect encrypted data to compress poorly because encryption makes it appear random. section 13.2.5
Encrypting a Kafka message
An application serializes a sensitive event as JSON, encrypts the serialized bytes in its producer, and publishes the ciphertext. Kafka stores and forwards the ciphertext. The consumer receives it, decrypts it, and then deserializes the original event. A serializer/deserializer wrapper supports whole-message encryption; encrypting individual fields requires a different approach. section 13.2.5
