concept producer in category cloud

This is an excerpt from Manning's book Designing Cloud Data Platforms.
Messages in streaming data systems are immutable. Once written by the producer they cannot be changed but a new version of the same message can be delivered later, so your data platform may need to be able to resolve multiple versions of the same message.
We’ll also go deeper into how streaming data is organized - with producers, consumers, messages, partitions and offsets. Then we’ll walk through some typical real-time data transformation use cases, wth particular attention on dealing with data deduplication, file format conversion, real-time data quality checks, and combining batch and real-time data.
Messages are written into the real time storage by
producersand are read and processed byconsumers. Both producers and consumers in this context are applications of some sort. As seen in Figure 6.6, producers write messages into topics (1) and consumers read from topics (2).Figure 6.7. Producers write messages into topics and consumers, which are applications, read from topics
![]()
So far, this doesn’t sound too different from a batch system where you can have file producers saving data to the storage and file consumers, like an ETL pipeline reading those files. But there are some fundamental differences in how producers can exchange data with multiple consumers.
In this real-time processing pipeline we have a single producer, which is our ingestion application, reading new, updated and deleted rows from some RDBMS. There is a single topic in our real-time storage and two different consumer applications. One (consumer Y) is a data transformation pipeline that performs common data transformations and saves processed messages to a different topic. Consumer Y is acting both as a consumer and as a producer, which is not uncommon in real-time processing systems. Another consumer, consumer X, is a real time analytics job of some sort. Maybe it’s a machine learning application, because otherwise why would it read raw data that hasn’t been processed by consumer Y yet. ;)
Messages in real-time storage systems are immutable. This means once a producer has written a message to the topic, it cannot change or delete it. As shown in Figure 6.7, when a message is written to the storage it is assigned an
offset (1)— an always increasing number that plays an important role in real-time processing systems. Offsets are used to send acknowledgements back to the producers, confirming that their message has been successfully saved (2). This way, producers know they can send a new message. Consumers also track offsets by sending the offset of the latest processed message back to the real time system. (3)Figure 6.8. Real-time storage systems provide a mechanism for tracking which messages have been written and which have been consumed by which application using a mechanism of offsets and acknowledgement.
![]()
The offset/acknowledgement mechanism is used for providing reliability. If a consumer fails (which it eventually will), it can easily resume processing where it left off by checking what was the last message offset it processed because producer offsets offer protection from failures in the real time storage system. If a producer didn’t receive an acknowledgement that a certain message has been successfully saved, it may decide to re-try saving the same message again.