A dead-letter queue (DLQ) is a Kafka topic where Kafka Connect sets aside records that a sink connector cannot deliver or process successfully. It lets the connector continue running while failed records remain available for investigation. section 5.5
Configure a DLQ alongside error handling in a sink connector:
- Set errors.tolerance to all so the connector continues through conversion, transformation, and sink-write errors. section 5.5
- Name the DLQ topic with errors.deadletterqueue.topic.name. section 5.5
- Enable context headers to preserve the reason for failure; storing the failed record alone does not explain the error. section 5.5
- Set the DLQ topic replication factor to 1 in a single-broker development cluster, or the connector may fail to start. section 5.5
Listing 5.4 REST call to install an elastic connector
$ curl -i -X PUT localhost:8083/connectors/
student-info-elasticsearch-connector/config \
-H "Content-Type: application/json" \
-d ‘{
"connector.class":
"io.confluent.connect.elasticsearch.ElasticsearchSinkConnector",
"connection.url": "http://elasticsearch:9200",
"tasks.max": "1",
"topics": "postgres_orientation_students", #1
"type.name": "_doc",
"value.converter": "org.apache.kafka.connect.json.JsonConverter",
"value.converter.schemas.enable": "false",
"schema.ignore": "true",
"key.ignore": "false",
"errors.tolerance":"all", #2
"errors.deadletterqueue.topic.name":"orientation_student_dlq", #3
"errors.deadletterqueue.context.headers.enable": "true", #4
"errors.deadletterqueue.topic.replication.factor": "1" #5
}’
Inspect the Elasticsearch sink connector configuration that enables error tolerance, names the DLQ topic, and preserves failure context in headers. section 5.5
