A commit has different meanings across systems. In a database, it finalizes a transaction, making its changes saved and visible. In Kafka, it usually confirms that messages are safely stored or records that a consumer has advanced beyond processed messages. section 1.4.3
For a Kafka consumer, committing records the offset of the next record to read. It marks the consumer’s position in the stream rather than individually acknowledging every message, allowing processing to resume after restart or failure. section 4.2
Offset commitment creates an architectural trade-off:
- Automatic commits simplify consumer code but can lose messages when an offset is committed before processing finishes. section 4.3.6
- Manual commits let the application commit only after processing completes, which is preferable for critical data. section 4.2.5 section 4.3.6
- If committed offsets are absent and consumption starts from the beginning, rereading can occur, so consumers must be idempotent. section 4.2.5
- Parallel workers can finish out of order; the commit may advance only to the highest contiguous offset whose processing is complete. section 4.3.1
Critical dashboard consumer
An operational-data dashboard disables automatic commits and manually commits offsets after processing. If offsets are missing, it reads from the beginning; repeated events therefore require idempotent handling. section 4.2.5
Figure 4.3 Workflow of the consumer application
Use the consumer workflow to trace polling, deserialization, processing, and offset commitment as one operating sequence. section 4.2
