chapter three

3 Using Fluentd to capture log events

 

This chapter covers

  • Configuring Fluentd for the input of log files,
  • Examining the impact of stopping and starting during file reading by Fluentd,
  • Extracting more meaning from log events by using parsers,
  • Performing self-monitoring and external monitoring of Fluentd using APIs.

With the conceptual and architecture foundations, setup, and have now run a simple configuration, we're ready to start looking at the capture of log events in more detail. In this chapter, we’re going to focus on capturing log events. But before we start that, we should look at how we can check that our Fluentd configuration is correct.

Setting up to follow and try the configurations

A quick note about how we’re presenting code in the book. To avoid the book becoming bloated with code and Fluentd configuration files, we’ve only included the configuration and code parts relevant to the subject being discussed. But the files referenced in the downloads and GitHub repository are the complete configurations (https://github.com/mp3monster/LoggingInActionWithFluentd). 

The repository includes both complete configurations, partial configuration files so you can implement configuration yourself. Along with this are scenarios and solutions to the scenarios that will allow you to try out your understanding of ideas in the book.

3.1  Dry running to check a configuration

3.1.1  Putting validating Fluentd configuration into action

3.2  Reading Log files

3.2.1  Putting the adaption of a Fluentd configuration to Fluent Bit into action

3.2.2  Rereading and resuming reading of log files

3.2.3  Configuration considerations for tracking position

3.2.4  Wild cards in the path attribute

3.2.5  Expressing time

3.2.6  Controlling the impact wild cards in filenames

3.2.7  Replacing wildcards with delimited lists in action

3.2.8  Handling log rotation

3.3  Self-Monitoring

3.3.1  HTTP interface check

3.4  Imposing structure on log events

3.4.1  Standard Parsers

rd Party Parsers

3.4.3  Applying a Regex parser to a complex log

3.4.4  Putting Parser configuration into action

3.5  Summary