4 Using Fluentd to output log events

Chapter 3 demonstrated how log events can be captured and how helper plugins such as parsers come into play. But capturing data is only of value if we can do something meaningful with it, such as delivery to an endpoint formatted so the log events can be used—for example, storing the events in a log analytics engine or sending a message to an operations (Ops) team to investigate. This chapter is about showing how Fluentd enables us to do that. We look at how Fluentd output plugins can be used from files, as well as how Fluentd works with MongoDB and collaboration/social tools for rapid notifications with Slack.

This chapter will continue to use the LogSimulator, and we will also use a couple of other tools, such as MongoDB and Slack. As before, complete configurations are available in the download pack from Manning or via the GitHub repository, allowing us to focus on the configuration of the relevant plugin(s). Installation steps for MongoDB and Slack are covered in appendix A.

4.1 File output plugin

4.1.1 Basic file output

4.1.2 Basics of buffering

4.1.3 Chunks and Controlling Buffering

4.1.4 Retry and backoff

4.1.5 Putting configuring buffering size settings into action

4.2 Output formatting options

4.2.1 out_file

4.2.2 json

4.2.3 ltsv

4.2.4 csv

4.2.5 msgpack

4.3 Sending log events to MongoDB