This chapter covers:
- Moving data and retention in Kafka
- Tools to help with data movement
- Data architectures Kafka enables
So far we have treated our data as moving into and out of Kafka for brief periods of time. Another decision to really analyze is where your data can live for longer terms. When we think of databases like MySQL or MongoDB, we do not always think if or how that data is expiring-we know it is for majority of your application’s entire lifetime. In comparison, Kafka storage can logically sit in between the practical storage solutions of a database and a message broker in regards to data storage; especially if we think of message brokers holding onto messages until they are consumed by a client as it is often in other message brokers.
As in all of engineering, tradeoffs are involved in each decision, including your business needs and requirements. Let’s look at a couple of options for storing and moving data in your environment.