This chapter covers:
- Inspecting requirements to determine how long to retain data.
- Investigating tools and options to help with data movement into and out of Kafka.
- Considering the data architectures Kafka enables such as Lambda and Kappa.
- Exploring storage choices for cloud and containers instances.
So far we have thought of our data as moving into and out of Kafka for brief periods of time. Another decision to analyze is where our data should live longer term. When you think of databases like MySQL or MongoDB, you may not always think about if or how that data is expiring. Rather, you know that the data is likely going to exist for the majority of your application’s entire lifetime. In comparison, Kafka storage logically sits somewhere between the long-term storage solutions of a database and the transient storage of a message broker; especially if we think of message brokers holding onto messages until they are consumed by a client as it is often in other message brokers.
Let’s look at a couple of options for storing and moving data in our environment.