3 The shipping stage: moving and storing telemetry
This chapter covers
- The role of the shipping stage in a telemetry pipeline
- Building emitter/shipper function in production code.
- Ways of moving telemetry through the Shipping Stage.
Figure 3.1: Telemetry pipeline stages with the Shipping stage as the second. The Shipping Stage receives telemetry prepared by the emitting stage and then processes and transforms it for storage. The Shipping Stage is the only stage that performs both markup and enrichment (see chapter 5).
The Shipping Stage in a telemetry pipeline is the second stage of the pipeline, as shown in figure 2.1. The Shipping Stage receives telemetry from the Emitting Stage, optionally marks up and enriches it (see chapter 6), and stores it for use in the Presentation Stage. When the Emitting Stage is entirely within your production code, the Shipping Stage can be combined directly into the Emitting Stage to make an emitter/shipper. Or the Shipping Stage might be a whole multi-system infrastructure handling the telemetry needs of your incredibly diverse production system. Both approaches are valid, and hybrid approaches equally so.
The Shipping Stage has two major roles it must perform:
- Move your telemetry between components of the Shipping Stage and ultimately into storage (chapters 3 and 4).
- Transform your telemetry through both markup and enrichment to add context (markup) and bring out details (enrichment) (chapter 6).