4 The Shipping stage: Unifying diverse telemetry formats

 

This chapter covers

  • Shipping locally emitted telemetry
  • Interacting with emitting stages with flexible and inflexible formats
  • Picking formats for shipping telemetry

This chapter is the second chapter covering the Shipping stage. Whereas chapter 3 covered moving telemetry around and preparing it for storage, this chapter covers the format (and format transformations) of telemetry data in the Shipping stage. A lot happens in the Shipping stage, which is why this stage is taking me a few chapters to describe:

  • Telemetry moves around (chapter 3). In larger systems, moving telemetry around can be quite complex, involving telemetry systems owned and controlled by different teams (multitenancy; chapter 7).
  • The Shipping stage needs to deal with whatever formats the Emitting stage can handle and in diverse environments (hardware and software), many formats (section 4.1).
  • The Shipping stage needs to not only move telemetry around, but also support any markup and enrichment (chapter 6) that happens during the Shipping stage and the formats those components need (section 4.2).

4.1 Shipping locally-emitted telemetry

4.1.1 Shipping telemetry from a log file

4.1.2 Shipping telemetry from the system logger

4.1.3 Shipping telemetry from standard output

4.2 Unifying diverse emitting formats

4.2.1 Encoding telemetry into strings

4.2.2 Picking a shipping format

4.2.3 Converting Syslog to JSON or other object-encoding formats

4.2.4 Designing with cardinality in mind

Summary