7 Filters

 

This chapter covers

  • creating content and attribute-based filtering.
  • modifying event content to enrich, structure, and help route events.
  • transforming content from logs to metrics.
  • rewriting tags to help downstream matching.
  • Grepping events to help identify events for filtering.
  • Limiting events to prevent backend overloading.
  • coding filters using Lua.

In the previous Chapter, we looked at Parsers and the Filter that enables the use of parsers. This chapter looks at the rest of the filters available to us. We will look carefully at filters that can manipulate the tag value, as this is central to how we route logs, metrics, and traces to the different output destinations.

We can see how Filters fit into our logical Fluent Bit architecture:

Figure 7.1 The application of filters we’re going to consider in this chapter is highlighted in our logical architecture.

We can group Filters in the following ways:

7.1 Integrate and enrich Filters

7.1.1 GeoIP – directing and securing logs

7.1.2 Checklist filter

7.2 Extend and amend with Filters

7.2.1 A brief look at the nest filter

7.2.2 Illustrating record_modifier filter

7.2.3 Illustrating modify filter

7.2.4 Bringing it together

7.2.5 Testing filters

7.3 Routing and control

7.3.1 Record accessor for tag rewriting and more

7.3.2 Rewrite tag filter example

7.3.3 Explicit event inclusion and exclusion with grep

7.4 Controlling events

7.4.1 Throttle

7.4.2 Log_to_metrics

7.4.3 Advanced use of matching

7.5 Custom filtering with Lua

7.5.1 Background to Lua

7.5.2 Implementing a Lua Filter

7.6 Summary

sitemap