Data transformation maps input events into another format or representation so systems can exchange, analyze, or further process data. In event-driven architectures, it is part of treating data as a governed contract across service boundaries, alongside validation and versioning. section 1.5
Choose the transformation location and processing model according to the architectural need:
- Use Kafka Connect Single Message Transformations for stateless, one-message-at-a-time changes such as type casting, filtering, and attribute renaming. Chains are configured statically and require connector redeployment when changed. section 7.3.4
- Use streaming applications for event-by-event processing, including mapping, filtering, routing, key changes, joins, windowing, and aggregations. section 8.0 section 8.3.1
- Use ETL when transformation belongs in a staging area and batch processing is acceptable; use ELT when the target system transforms data after loading it in raw form. section 8.1.1
Customer 360 aggregation
An intermediate streaming service can join transactions with customer profiles and publish an aggregated result to Kafka. A final service then reads prepared data instead of performing the joins itself, making the derived result available to other consumers. section 8.0
Figure 7.20 Transformations are stateless and can be chained together.
Use the diagram to compare stateless transformation stages and their chained flow through Kafka Connect. section 7.3.4
