Log rotation keeps long-running log files from becoming too large and unwieldy by moving or replacing files while new events continue elsewhere. It matters when collectors and servers start at different times: the collector must avoid both missed events and reingestion after a restart. section 3.5.2
For Fluent Bit tail input, persist read progress in the DB setting. The SQLite-backed tracking database records how far ingestion has advanced and which files were partly or fully consumed, supporting recovery without replaying prior logs. section 3.5.2
A rotation-aware tail configuration combines:
- A file path and read_from_head setting to select and begin reading the log. section 3.5.2
- A DB path to preserve ingestion progress across Fluent Bit restarts. section 3.5.2
- rotate_wait and ignore_older to handle rotated files and skip sufficiently old ones. section 3.5.2
Rotation-aware file tail
Use a tail input with DB ./tracking-file-read.db, rotate_wait 1, and ignore_older 1m. This pairs persistent offsets with a brief rotation wait and an age limit for old files. section 3.5.2
Figure 3.4 We can see how log rotation moves and then deletes log files.
Inspect the file lifecycle: rotation moves a log file and later deletes it, illustrating why the collector must track progress across changing files. section 3.5.2
