Data transformation maps an incoming event to a representation that downstream calculations can use. In a platform metrics pipeline, event sources supply CloudEvents; transformation functions turn selected events into CDEvents, which are then used to calculate metrics. section 9.2.2
Keep collection, transformation, and calculation separate:
- Adapt sources that do not emit CloudEvents before their data enters the pipeline. section 9.2.2
- Store incoming CloudEvents in PostgreSQL. Periodically fetch unprocessed events rather than blocking to transform each one on arrival. section 9.2.2 section 9.2.5
- Route events to single-purpose transformation functions and store the resulting CDEvents separately for metric calculations. Functions can scale with traffic or downscale when unused. section 9.2.2 section 9.2.5
Map a deployment event
A CloudEvent whose type is dev.knative.apiserver.resource.add and whose data.InvolvedObject.Kind is Deployment maps to a dev.cdevents.service.deployed.0.1.0 CDEvent. This mapping supplies a deployment signal for metrics such as deployment frequency; other metrics require additional transformations. section 9.2.5 section 9.2.7
Figure 9.19 Data flows from Data Sources producing CloudEvents to the CloudEvents endpoint whose only mission is to store these events into the Event Store. From there, the CloudEvents Router have the logic to decide where to route events to transformation functions, which allows us to map CloudEvents to CDEvents for further processing. Once we have CDEvents, the Calculation Functions can read these events to aggregate data and produce metrics. Metrics consumers can get the metrics by interacting with the Metric Endpoint, which will fetch the calculated metrics from the metrics database.
Follow the stored CloudEvent through routing and transformation to the CDEvent consumed by a calculation function. section 9.2.7
