A Single Message Transform (SMT) modifies each Kafka Connect record independently as it moves between a connector and its converter. A source connector transforms records before conversion into Kafka; a sink connector transforms converted records before delivery to an external system. This makes SMTs useful for lightweight format changes without changing the producer or consumer application. section 5.1 section 5.4
Use SMTs when the transformation can be applied to one record at a time:
- Mask sensitive fields before customer data enters Kafka or before records leave it. section 5.1 section 1.5
- Filter records, flatten nested structures, remove fields, or adjust record keys and values. section 5.4
- Keep joins, aggregations, and other complex processing in Kafka Streams or ksqlDB rather than an SMT. section 5.4
Masking and extracting a customer key
Configure a connector chain named copyFieldToKey, extractKeyFromStruct, and maskSsn. ValueToKey copies user_name into the key, ExtractField$Key extracts user_name from the resulting Struct, and MaskField$Value replaces ssn with xxx-xx-xxxx. Transform names execute in the order listed, so each step must account for the record shape produced by the previous step. section 5.4
Listing 5.2 The JSON transform configuration
"transforms":"copyFieldToKey, extractKeyFromStruct, maskSsn", #1
"transforms.copyFieldToKey.type":
"org.apache.kafka.connect.transforms.ValueToKey", #2
"transforms.copyFieldToKey.fields":"user_name", #3
"transforms.extractKeyFromStruct.type":
"org.apache.kafka.connect.transforms.ExtractField$Key", #4
"transforms.extractKeyFromStruct.field":"user_name", #5
"transforms.maskSsn.type":
"org.apache.kafka.connect.transforms.MaskField$Value", #6
"transforms.maskSsn.fields":"ssn", #7
"transforms.maskSsn.replacement":"xxx-xx-xxxx" #8
Inspect the complete JSON configuration for ordered ValueToKey, ExtractField, and MaskField transforms. section 5.4
