4 Creating consumer applications

Here we’ll shift from producers to the other half of the pipeline: consumers. We need to understand how they read at scale, coordinate, and stay correct. How are messages received and processed in parallel, how do applications subscribe to topics or explicitly position themselves in a stream, and how do batching and timeouts shape throughput and latency? To answer these key questions, we need to make consumer groups concrete, explore rebalances, and highlight the most common problems, which are lag, duplicates, and ordering. We also want to consider Confluent REST Proxy as a lightweight integration option. By the end of this chapter, you’ll know how to design consumer logic that is reliable, efficient, and easy to operate.

In our ODS example, we’ll focus on the next stage of the project: developing the consumer application. With topics set for customer profile data and transactions, the Customer 360 team will now delve into the intricacies of data aggregation, determining how best to handle and merge the incoming streams. After discussing the various trade-offs, the next step is to start designing the consumer application, focusing on data aggregation and processing it in the client code.

4.1 Organizing consumer applications

4.2 Receiving a message

4.2.1 Reading data in parallel

4.2.2 Setting initial consumer configuration for the Customer 360 project

4.2.3 Group leader and group coordinator

4.2.4 Committing the offsets

4.2.5 Specifying the strategy for committing offsets for an ODS

4.2.6 Creating batches

4.2.7 Timeouts and partition rebalance

4.2.8 Static group membership

4.2.9 Partition assignment strategies

4.2.10 The next-gen consumer rebalance protocol

4.2.11 Subscriptions and assignment

4.2.12 Reading data from compacted topics

4.2.13 Consumer considerations for the ODS project

4.3 Common consumer problems

4.3.1 Consumer scalability challenges

4.4 Data compression