chapter three

3 Using Fluentd to capture log events

 

This chapter covers

  • Configuration of log file inputs
  • Configuration to handle log file rotation
  • The impact of stop and start during file reading
  • Parsing log events
  • Using parsers to get more meaning out of log events
  • Self-monitoring and the API for remote monitoring

With the conceptual and architecture foundations, setup, and having run a very simple configuration, we’re ready to start looking at the capture of log events in more detail. Within the book, we are limiting the code shown to the segments that are significant to the matter in hand. However, the complete configuration can be seen if you look at the file being referenced in the downloads. If you’re enthusiastic you can start by using the Chapter3/Fluentd/no-source-config.conf configuration file and incorporate the new pieces into it as we go, otherwise follow along with the complete configurations. Also, we have provided some exercises and questions, so as you progress through the chapters you can test your understanding. The answers to the questions and exercises are provided at the end of each chapter, along with the downloads including the worked answer of the exercises.

3.1    Dry running to check a configuration

3.1.1   Scenario – validating Fluentd configuration

3.2    Reading Log files

3.2.1   Scenario - Adapt Fluentd configuration for Fluent Bit

3.2.2   Rereading and resuming reading of log files

3.2.3   Configuration considerations for tracking position

3.2.4   Wild cards in the path attribute

3.2.5   Expressing time

3.2.6   Controlling the impact wild cards in filenames

3.2.7   Exercise - Replacing wildcards with delimited lists

3.2.8   Handling log rotation

3.3    Self-Monitoring

3.3.1   HTTP interface check

3.4    Imposing Structure on log events

3.4.1   Standard Parsers

3.4.2   3rd Party Parsers

3.4.3   Applying a Regex parser to a complex log

3.4.4   Exercise - Configuring a Parser

3.5    Summary