chapter six

6 Parsing

 

This chapter covers

  • Exploring the relationship between filters, parsers, and decoders.
  • Examining the prebuilt parsers and how they work
  • Using filters to run parser processes
  • Implementing REGEX and JSON parsers to extract meaning from log content.

In this chapter, we will start working with Fluent Bit’s capabilities to examine and manipulate the data it collects and outputs. As we can see in the following figure, this means that parsers and filters sit within the middle of the pipeline of processing log events after we’ve ingested the data.

Figure 6.1 Logical architecture of Fluent Bit, with this Chapter's focus on parsers highlighted.

The benefit of parsers being separated from the input and output plugins is that we can apply the same parsing processes to different sources. This makes parsers highly reusable, and as we’ll see.

6.1 The goal of parsing

Parsers are one of the most important tools when it comes to extracting meaning from log events. The use of parsers allows us to find important values. The main reasons for parsing are:

6.2 Relationship between Parsers and Filters

6.3 Prebuilt Parsers

6.4 Parsing an Apache log file.

6.5 Custom parsing

6.5.1 Handling Escape Characters

6.5.2 Parsing when dealing with multiline log files.

6.6 Processing JSON

6.6.1 Diagnosing the Unhappy Paths

6.7 Other types of parsers

6.7.1 Logfmt

6.7.2 ltsv

6.8 Decoders

6.9 Parsing shortcut for file inputs

6.10 Summary