Backpressure occurs when an upstream producer emits events faster than a downstream component can consume them. In Fluent Bit pipelines, sudden network-input spikes or fast stdout producers can leave the source blocked until events are consumed. section 1.2.2 section 3.3.2
Reduce backpressure risk by choosing an appropriate ingestion path and limiting work at the constrained point:
- Keep stdin parsing simple, enable buffering, and move expensive processing to a downstream node. section 3.3.2
- Prefer writing producer output to a file and collecting it with the tail plugin when direct piping is unsafe. section 3.3.2
- Manage growing log files with logrotate or an equivalent Windows utility. section 3.3.2
A fast stdout pipeline
A command such as ./SimulatorConfig/stdout-formatted-run.sh | fluent-bit can expose backpressure when the simulator produces output faster than Fluent Bit consumes it. The source may block, so buffering and simple parsing are safer than performing heavy work inline. section 3.3.2
Figure 1.2 Logical Fluent Bit architecture, with the blocks representing logical features and the lines representing the possible flow of events. The standard Fluent Bit groupings are overlaid, but I’ve separated and illustrated the buffers slightly differently, as their positions are more logical than how they fit into the code base.
Use the logical Fluent Bit architecture to trace where listener inputs, pulled inputs, parsers, buffers, and downstream processing can constrain event flow. section 1.2.2
