chapter one

1 Introduction to Fluentd

 

This chapter covers

  • Examing the range of use cases that logs, log events can support, along with how Fluentd and Fluent Bit can help,
  • Identifying the value of log unification,
  • Differentiating between log analytics and unified logging,
  • Illustrating the current monitoring concepts such as the 4 golden pillars,
  • Understanding Fluentd's evolution and adoption,
  • Highlighting the differences between Fluentd and Fluent Bit.

Before getting into the detail of Fluentd, we should take the time to first focus on the motivations for using a tool such as Fluentd. How can logging help us? What is log analytics, and why is log unification necessary? These are among the questions we will work to answer in this chapter. We'll highlight the kinds of activities logging can help or enable us to achieve.

Let's also take a step back and understand some contemporary thinking around how systems are measured and monitored; understanding these ideas will mean we can use our tools more effectively. After all, a tool is only as good as the user creating the configuration or generating log events to be used.

As we do this, it is worth taking time to understand how Fluentd has evolved and why it holds its position within the industry. If you are considering Fluentd as a possible tool or looking to make a case for its adoption, then it is helpful to understand the 'origins story' as this will inform how Fluentd may be perceived.

1.1  Elevator pitch for Fluentd

1.2  Why do we produce logs?

1.3  Evolving ideas

1.3.1  Four Golden Signals

1.3.2  Three Pillars of Observability

1.4  Log unification

1.4.1  Unifying logs vs log analytics

1.5  Software stacks

1.5.1  ELK stack

1.5.2  Fluentd – Logstash comparison

1.5.3  The relationship between Fluentd and Fluent Bit

1.5.4  The relationship between Logstash and Beats

1.6  Log routing as a vehicle for security

1.7  Log Event Lifecycle

1.8  Evolution of Fluentd

1.8.1  Treasure Data

1.8.2  CNCF

1.8.3  Relationship to major cloud vendors PaaS/IaaS

1.9  Where can Fluentd and Fluent Bit be used

1.9.1  Platform constraints

1.10  Fluentd UI based editing

1.11  Plugins

1.12  How Fluentd can be used to make operational tasks easier

1.12.1  Actionable log events

1.12.2  Making logs more meaningful

1.12.3  Polyglot environments

1.12.4  Multiple targets

1.12.5  Controlling log data costs

1.12.6  Logs to metrics