chapter one

1 Introduction to Fluentd

 

This chapter covers

  • The range of use cases that logs can support
  • The value of log unification.
  • Differentiating between log analytics and unified logging.
  • Illustrating the current monitoring concepts such as the 4 golden pillars.
  • Understanding Fluentd’s evolution and adoption.
  • Highlighting the differences between Fluentd and Fluent Bit.

Before getting stuck into the workings of Fluentd, this chapter is going to focus on the motivations for using a tool such as Fluentd. How can logging help us, what are log analytics and why is log unification important are among the questions we will work to answer in this chapter. We’ll highlight the kinds of activities logging can help or enable us to achieve.

Let’s also take a step back and understand some contemporary thinking around how systems are measured and monitored. After all, a tool is only as good as the people using it (both in terms of incorporating log events in the code during development and the operational use of the logs). So we will take a moment to touch upon the ideas, we’ll be able to make better use of our tool.

1.1    Elevator pitch for Fluentd

1.2    Why do we produce logs?

1.3    Evolving ideas

1.3.1   Four Golden Signals

1.3.2   Three Pillars of Observability

1.4    Log unification

1.4.1   Unifying logs vs log analytics

1.5    Software stacks

1.5.1   ELK Stack

1.5.2   Fluentd – Logstash comparison

1.6    Log routing as a vehicle for security

1.7    Log Event Lifecycle

1.8    Evolution of Fluentd

1.8.1   Treasure Data

1.8.2   CNCF

1.8.3   Relationship to major cloud vendors PaaS/IaaS

1.9    The relationship between Fluentd and Fluent-Bit

1.10 Where can it be used

1.10.1    Platform constraints

1.11 Fluentd configuration UI based editing

1.12 Plugins

1.13 How Fluentd can be used to make operational tasks easier

1.13.1    Actionable Log Events

1.13.2    Making Logs More Meaningful

1.13.3    Polyglot Environments

1.13.4    Multiple Targets

1.13.5    Controlling Log data costs

1.13.6    Logs to metrics

1.13.7    Rapid operational consolidation

1.14 Summary