Apache Kafka is a distributed streaming platform for real-time data pipelines and streaming applications. It accepts streams from multiple sources and delivers them to multiple consumers, serving as a middle layer for real-time data exchange. Its append-only, immutable data model suits substantial real-time or near-real-time event volumes, not record updates, deletions, or complex queries. section 5.2 section 5.4
A practical use is asset-location monitoring: an external system emits movement events, Kafka receives them on the asset_location topic, and a consumer turns each event into a domain update. Keep the consumer focused on translating and publishing events rather than directly reaching into a central application object; a Mediator can route AssetLocationUpdated events to the appropriate handler and reduce coupling. section 5.2 section 5.4
Listing 5.13 Docker Compose file to launch Kafka with Zookeeper
version: '2.1'
services:
zookeeper:
image: confluentinc/cp-zookeeper:7.3.2
container_name: zookeeper
ports:
- "2181:2181"
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_SERVER_ID: 1
ZOOKEEPER_SERVERS: zoo1:2888:3888
kafka:
image: confluentinc/cp-kafka:7.3.2
hostname: kafka
container_name: kafka
ports:
- "9092:9092"
- "29092:29092"
- "9999:9999"
environment:
KAFKA_ADVERTISED_LISTENERS:
INTERNAL://kafka:19092,EXTERNAL://
${DOCKER_HOST_IP:127.0.0.1}:9092,
DOCKER://host.docker.internal:29092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT,
DOCKER:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
KAFKA_ZOOKEEPER_CONNECT: "zookeeper:2181"
KAFKA_BROKER_ID: 1
KAFKA_LOG4J_LOGGERS: "kafka.controller=
INFO,kafka.producer.async
.DefaultEventHandler=INFO,
state.change.logger=INFO"
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
KAFKA_JMX_PORT: 9999
KAFKA_JMX_HOSTNAME: ${DOCKER_HOST_IP:-127.0.0.1}
KAFKA_AUTHORIZER_CLASS_NAME: kafka.security.authorizer.AclAuthorizer
KAFKA_ALLOW_EVERYONE_IF_NO_ACL_FOUND: "true"
depends_on:
- zookeeper
Run Kafka locally with Docker Compose alongside Zookeeper. Treat generated setup as a starting point: verify image versions and hardware compatibility rather than assuming an AI-generated configuration is current. section 5.2
Work with a local topic through the Kafka console:
- List topics, then create asset_location with one partition and replication factor 1. section 5.2
- Produce JSON location messages containing an asset ID, latitude, longitude, and timestamp. section 5.2
- Consume messages from the beginning to inspect the stream, then delete the topic when the experiment is complete. section 5.2
Location-update consumer
A Python KafkaConsumer subscribes to asset_location, deserializes each JSON message, and constructs an AssetLocationUpdated event. Publishing that event through a mediator lets AssetManager register its update handler without making the Kafka adapter depend directly on AssetManager. Test with several timestamped location messages and confirm that each produces the expected location history update. section 5.2
