In an asynchronous Kafka pipeline, a missing business event is not necessarily a broker failure. It may never have been sent, may have gone unacknowledged, or may have been skipped after a consumer committed its offset. Because a missing event can go undetected, durability must be assessed across the whole path from producer to final processing. section 7.4 section 4.3.6
Check where the event could have left the intended flow:
- Before or within Kafka: the producer might not send it, an unhandled missing acknowledgment might conceal a failed write, or a leader might fail before replication. section 7.4
- While stored: a retention policy might delete the message before a consumer reads it. section 7.4
- During processing: a consumer might commit an offset before finishing the associated work. After a failure or rebalance, it resumes at the next offset and skips that work. section 4.3.6 section 7.4
A connector that keeps running
Suppose a Kafka Connect sink cannot write a message because its schema conflicts with the target system. Redirecting that message to a dead-letter topic lets the connector continue, but the message is absent from the main flow. Operations must monitor the dead-letter topic and decide how to reapply its messages; restoring their original order may be difficult, and reconciliation or a full reload may be necessary. A source connector facing an incompatible schema instead may stop, requiring prompt detection. section 7.3.10
Set the loss tolerance before choosing controls. Critical events allow no message loss and require persistence and full acknowledgment; less critical flows may accept occasional loss with fallback mechanisms or best-effort delivery. Commit consumer offsets only after processing when skipped work is unacceptable. Define monitoring, alerts, and recovery for messages diverted from normal processing. Tracing each pipeline step can help locate a suspected loss, but tracing high volumes has a resource cost. section 10.1.4 section 4.3.6 section 7.3.10 section 7.4
