4 Using Fluentd to output log events

In Chapter 3, we saw how log events can be captured and how helper plugins such as parsers come into play. But capturing data is only of value if we can do something meaningful with it, such as delivery to an endpoint formatted so the log events can be used, for example, stored in a Log Analytics engine or sent as a message to an ops team of investigation. This chapter is about showing how Fluentd enables us to do that. Now that we have configured some source inputs, this chapter explores how to get the data back out. We look at how Fluentd can meet some of the claims made in Chapter 1, such as the value of getting important events to notification mechanisms rather than waiting for events to be aggregated and periodically analyzed.

This chapter will continue to use the Log Simulator, and we will also use a couple of other tools such as MongoDB and Slack. As before, complete configurations are available in the download pack from Manning or via the GitHub repository, allowing us to focus on the configuration of the relevant plugin(s).

4.1 File Output Plugin

4.1.1 Basic file output

4.1.2 Basics of buffering

4.1.3 Chunks & Controlling Buffering

4.1.4 Retry and back off

4.1.5 Putting Configuring Buffering Size Settings into action

4.1.6 Output formatting options

4.1.7 Applying Formatters

4.1.8 Putting Configuring JSON Formatter into action

4.2 Sending log events to MongoDB

4.2.1 Deploying MongoDB Fluentd plugin

4.2.2 Configuring the Mongo output plugin for Fluentd

4.2.3 Exercise - Define a MongoDB Connection string

4.3 Actionable log events

4.3.1 Actionable Log Events through service invocation

4.3.2 Actionable through user interaction tools

4.4 Slack to demonstrate the social output

4.4.1 Handling tokens and credentials more carefully

4.4.2 Exercise - Externalizing Slack configuration attributes

4.5 The right tool for the right job

4.6 Summary