4 Agents and instrumentation

 

This chapter covers

  • What we mean by agents and instrumentation
  • What types of agents exist, for what signal types
  • What OpenTelemetry is and how you can benefit from it
  • Criteria for selecting an agent

In this chapter, we will focus on how to get the signals from the sources we discussed in chapter 3 to the destinations, which we will be focusing on in chapters 5 and 6. Since observability is all about getting actionable insights from the signals the systems under observation expose, without the agents and instrumentation, there would be little to observe.

In a nutshell, in this chapter, we will learn how to instrument code (and automate that task) as well as select and deploy agents that collect, aggregate, filter, downsample, redact, and route logs, metrics, and traces.

The telemetry industry is, at the time of writing, in the midst of a tectonic shift. This transformation from vendor-specific or signal-specific instrumentation and agents to an industry standard called OpenTelemetry started in 2019. In the context of this book, we consider vendor-specific as well as signal-specific agents as traditional agents, in contrast to OpenTelemetry.

4.1 Log routers

4.1.1 Fluentd and Fluent Bit

4.1.2 Other log routers

4.2 Metrics collection

4.2.1 Prometheus

4.2.2 Other metrics agents

4.3 OpenTelemetry

4.3.1 Instrumentation

4.3.2 Collector

4.4 Other agents

4.5 Selecting an agent

4.5.1 Security for and of the agent

4.5.2 Agent performance and resource usage

4.5.3 Agent nonfunctional requirements