A filter is a Fluent Bit pipeline component that fetches received events, applies logic to them, and returns the resulting events to the buffer. Filters are the main place to manipulate event content after ingestion: enrich records, reshape attributes, control flow, change tags for routing, or derive metrics from logs. section 1.2.2 section 7.0
Figure 7.1 The application of filters we’re going to consider in this chapter is highlighted in our logical architecture.
Filters sit in the processing path after events enter the pipeline, where they can alter records before downstream outputs consume them. section 7.1
Choose filters by the operational result needed:
- Integrate and enrich existing events with contextual data or external evaluation, such as Kubernetes metadata, AWS instance details, GeoIP2 locations, sensitive-data detection, or TensorFlow classification. section 7.1
- Extend and amend payloads by parsing fields, conditionally renaming, copying, or removing attributes, adding or selecting fields, and reorganizing nested structures. section 7.1 section 7.7
- Route and control traffic by rewriting tags for downstream matching, throttling throughput to protect backends, or converting logs into metrics. section 7.0 section 7.7
Tags are especially important because changing a tag changes how later match rules route logs, metrics, and traces to output destinations. Tag-rewrite rules determine which events receive changed tags, while throttling limits event flow when a backend could be overloaded. section 7.0 section 7.7
Most filters are configured declaratively. When built-in behavior is insufficient, small custom processes can use Lua; more demanding custom filters can be implemented with C, Go, or WebAssembly as precompiled extensions. section 1.2.2 section 1.3.3
