3 The Shipping stage: Moving and storing telemetry

 

This chapter covers

  • The role of the shipping stage in a telemetry pipeline
  • The emitter/shipper function in production code
  • Ways of moving telemetry through the Shipping stage

The Shipping stage in a telemetry pipeline is the second stage of the pipeline, as shown in figure 3.1. The Shipping stage receives telemetry from the Emitting stage, optionally marks up and enriches it (see chapter 6), and stores it for use in the Presentation stage. When the Emitting stage is entirely within your production code, the Shipping stage can be combined directly with the Emitting stage to make an emitter/shipper, or the Shipping stage might be a whole multisystem infrastructure handling the telemetry needs of your incredibly diverse production system. Both approaches are valid, and hybrid approaches are equally so.

Figure 3.1 Telemetry pipeline stages, with the Shipping stage second. The Shipping stage receives telemetry prepared by the Emitting stage and then processes and transforms it for storage. The Shipping stage is the only stage that performs both markup and enrichment (chapter 6).

The Shipping stage must perform two major roles:

  • Move telemetry between components of the Shipping stage and ultimately into storage (chapters 3 and 4)
  • Transform telemetry through both markup and enrichment to add context (markup) and bring out details (enrichment) (chapter 6)

3.1 Emitter/shipper functions, telemetry from production code

3.1.1 Shipping directly into storage

3.1.2 Shipping through queues and streams

3.1.3 Shipping to SaaS systems

3.2 Shipping between SaaS systems

3.3 Tipping points in Shipping-stage architecture

Summary

sitemap