When a Kafka operation fails or times out, retry repeats the send or connector operation after a delay instead of failing immediately. The Kafka producer handles retries within configured time limits, while Kafka Connect can retry access to external systems. section 8.1.3 section 11.6.2
A retry policy combines several controls:
- The producer waits for a broker response up to request.timeout.ms, then waits retry.backoff.ms before retrying. section 8.1.3
- delivery.timeout.ms limits the total delivery period; after it expires, the producer gives up and throws an exception. section 8.1.3
- Kafka Connect uses errors.retry.timeout and errors.retry.delay.max.ms for failed external-system operations; connector-specific settings may add attempt counts and backoff intervals. section 11.6.2
- Retries should not become blind application-level resends: producer-library retries preserve ordering, particularly when enable.idempotence is enabled, while a circuit breaker can intercept persistent failure. section 8.1.3
Temporary database outage
A JDBC connector normally attempts a failed database connection three times at 10-second intervals. During maintenance or a short network outage, increasing connection.attempts and connection.backoff.ms can provide more recovery time and avoid a connector restart. section 11.6.2
Figure 8.2 We can define numerous timeouts in the producer. The most important are linger.ms, which we can use to configure batching, and delivery.timeout .ms, which we can use to configure when the Kafka library stops redelivering messages and throws an error.
Use the timeout diagram to relate batching delay, broker-response timeout, retry backoff, and the total delivery deadline. section 8.1.3
