Suppose customer names are kept in a file, but applications need them in a Kafka topic. A Kafka Connect connector defines that transfer without requiring a custom producer or consumer. A source connector reads from an external system into Kafka; a sink connector reads from Kafka and writes to an external system. section 11.0 section 11.6.1
{
"name": "customers-source",
"config": {
"connector.class": "FileStreamSource",
"tasks.max": "1",
"file": "/tmp/customers.txt",
"topic": "customers"
}
}
This JSON names a FileStreamSource connector and sets its class, task limit, input file, and destination topic. Submit it with `curl -X POST -H "Content-Type: application/json" --data @source-connector.json http://localhost:8083/connectors`. FileStreamSource is useful for this demonstration but is not recommended for production. section 11.2.2
A connector divides its work into tasks, which Kafka Connect runs on workers. `tasks.max` sets an upper bound, not a guarantee of parallelism: one local file cannot usefully supply multiple tasks. In Distributed Mode, workers with the same `group.id` redistribute tasks when a worker joins or fails. The local-file example does not gain that resilience if another worker cannot access the file. section 11.3 section 11.6.1
Check the file-to-topic flow
Request `GET /connectors/customers-source/status` to check the connector and task states. Append lines to `/tmp/customers.txt`, then consume the `customers` topic to verify their arrival. If nothing appears, check the status endpoint and worker logs. On restart, this connector remembers its file position and reads new appended lines; edits to lines it already read do not produce new messages. section 11.2.3
The REST API also supports changing a connector’s configuration, pausing it during external-system maintenance, resuming it, and restarting failed tasks after a problem is fixed. Choose a connector suited to the external system and its failure modes rather than treating the minimal file connector as a production default. section 11.5.2 section 11.2.2
