chapter one

1 Introduction to Fluentd

 

This chapter covers

  • Examing the range of use cases that logs can support,
  • Identifying the value of log unification,
  • Differentiating between log analytics and unified logging,
  • Illustrating the current monitoring concepts such as the 4 golden pillars,
  • Understanding Fluentd's evolution and adoption,
  • Highlighting the differences between Fluentd and Fluent Bit.

Before getting into the detail of Fluentd, we should take the time to first focus on the motivations for using a tool such as Fluentd. How can logging help us? What is log analytics, and why is log unification necessary? These are among the questions we will work to answer in this chapter. We'll highlight the kinds of activities logging can help or enable us to achieve.

Let's also take a step back and understand some contemporary thinking around how systems are measured and monitored. After all, a tool is only as good as the user creating the configuration or generating log events to be used. So we will take a moment to touch upon the ideas, we'll be able to make better use of our tool.

As we do this, it is worth taking time to understand how Fluentd has evolved and why it holds its position within the industry. If you are considering Fluentd as a possible tool or looking to make a case for its adoption; then it is helpful to understand the 'origins story' as this helps inform how Fluentd may be perceived, where it might be in terms of adoption

1.1 Elevator pitch for Fluentd

1.2 Why do we produce logs?

1.3 Evolving ideas

1.3.1 Four Golden Signals

1.3.2 Three Pillars of Observability

1.4 Log unification

1.4.1 Unifying logs vs. log analytics

1.5 Software stacks

1.5.1 ELK Stack

1.5.2 Fluentd – Logstash comparison

1.5.3 The relationship between Fluentd and Fluent-Bit

1.5.4 The relationship between Logstash and Beats

1.6 Log routing as a vehicle for Security

1.7 Log Event Lifecycle

1.8 Evolution of Fluentd

1.8.1 Treasure Data

1.8.2 CNCF

1.8.3 Relationship to major cloud vendors PaaS/IaaS

1.9 Where can Fluentd and Fluent bit be used

1.9.1 Platform constraints

1.10  Fluentd configuration UI based editing

1.11  Plugins

1.12  How Fluentd can be used to make operational tasks easier

1.12.1 Actionable Log Events

1.12.2 Making Logs More Meaningful

1.12.3 Polyglot Environments

1.12.4 Multiple Targets

1.12.5 Controlling Log data costs