System integration coordinates separate systems so they can exchange data or functionality. In Kafka architectures, Kafka can act as a central hub for streaming data between enterprise systems, but its fit depends on requirements, alternatives, and operational trade-offs. section 5.0 section 7.0
A data-first approach treats messages as contracts across service boundaries. Reliable integration therefore requires attention to message structure, validation, versioning, transformations, schema governance, and the placement of data movement within the architecture. section 1.5
Architectural choices include:
- Use Kafka Connect to move data into and out of Kafka, while recognizing security, data-contract, and operational concerns. section 7.0 section 7.3.9
- Prefer existing products when they fit; RabbitMQ, Pulsar, and managed cloud services may be more appropriate for particular integration needs. section 5.0
- Govern asynchronous transfers through durability, ordering, exactly-once processing, schemas, and controlled transformations. section 7.0 section 1.5
Customer 360 operational data store
An operational data store can use a JDBC source connector to load immutable transaction rows from a database table into the TRANSACTIONS Kafka topic. An incrementing ID tracks new records, while Avro conversion and Schema Registry support structured messages. A separate Kafka Connect cluster provides safer isolation than exposing the database model directly to the integration layer. section 7.3.9
Listing 7.1 Configuration of the JDBC source connector
{ "name": "jdbc-source-transactions",
"config":
{"connector.class": "JdbcSource", #1
"tasks.max": 1, #2
"topic.prefix": "connect-", #3
"connection.url": "jdbc:postgresql://postgres:5432/trans_db", #4
"mode": "incrementing", #5
"incrementing.column.name":"ID", #6
"value.converter": "io.confluent.connect.avro.AvroConverter", #7
"value.converter.schema.registry.url":"http://schema-registry:8081", #8
"table.whitelist" : "public.transactions", #9
"connection.user": "demo",
"connection.password" : "demo"
}
}
Inspect the JDBC source connector configuration, including the database connection, incrementing mode, ID column, topic prefix, Avro converter, Schema Registry, and table selection. section 7.3.9
