4 Creating Consumer Applications

Here we shift from producers to the other half of the pipeline: consumers. We need to understand how they read at scale, coordinate, and stay correct. This chapter explains how messages are received and processed in parallel, how applications subscribe to topics or explicitly position themselves in a stream, and how batching and timeouts shape throughput and latency. We make consumer groups concrete, explore rebalances, and highlight the most common issues (lag, duplicates, ordering). We also touch on Kafka REST Proxy as a lightweight integration option. By the end, you’ll know how to design consumer logic that is reliable, efficient, and easy to operate.

4.1 Field notes: Consumer patterns and trade-offs

Gathered once more in their meeting room, Max Sellington, Rob Routine, and Eva Catalyst focus on the next stage of their project: developing the consumer application. With topics set for customer profile data and transactions, they now delve into the intricacies of data aggregation, determining how best to handle and merge the incoming streams.

4.2 Organizing consumer applications

4.3 Receiving a message

4.3.1 Reading data in parallel

4.3.2 Field notes: Setting initial consumer configuration for Customer 360 project

4.3.3 Group Leader and Group Coordinator

4.3.4 Committing the offsets

4.3.5 Field notes: Specifying the strategy for committing offsets for Customer360

4.3.6 Creating batches

4.3.7 Timeouts and partition rebalance

4.3.8 Static Group Membership

4.3.9 Partition assignment strategies

4.3.10 The Next-Gen Consumer Rebalance Protocol

4.3.11 Subscriptions and assignment

4.3.12 Reading data from compacted topics

4.3.13 Field notes: Consumer considerations for the Customer360 project

4.4 Common consumer issues

4.5 Data compression

4.6 Accessing Kafka through REST Proxy

4.7 Online resources

4.8 Summary