SQLite is a small-footprint database implemented in C and built into the Fluent Bit binary. In file-tail inputs, it records ingestion progress: how far Fluent Bit has read through a log file and which files are partially or fully ingested. This persistent state lets a restarted collector resume work without reingesting earlier log events from long-running processes. section 3.5.2 section 3.9
Configure the tail input with `DB` to name the database file. Fluent Bit uses that file for tracking data, while SQLite manages storage at a low level; its optimized I/O can be faster than maintaining equivalent state in text files. The same local-state pattern is useful where connectivity may be unavailable. section 3.5.2
Use the database settings to balance recovery, isolation, and concurrent access:
- Set `DB` to a tracking-file path for each tail input that needs durable read-position state. section 3.5.2
- Prefer one Fluent Bit process updating a database at a time; `DB.locking` controls whether multiple processes may update its file. section 3.5.2
- Give each log source its own database file to isolate tracking state. section 3.5.2
- Account for extra file handles when workers parallelize processing, so database read/write management does not reduce worker benefits. section 3.5.2
- Query the database from other processes when inspection is needed, even while Fluent Bit uses it for tracking. section 3.5.2
Track a tailed file across restarts
A tail input can read `./basic-file.txt` from its beginning, write its progress to `./tracking-file-read.db`, and emit matching records to stdout. With `DB.locking false`, the configuration explicitly sets the database locking behavior. Running the collector again can use the stored progress rather than treating the file as entirely new input. section 3.5.2
Listing 3.5 An enhanced file tail: basic-file-read.conf
[SERVICE] flush 1 [INPUT] name tail path ./basic-file.txt DB ./tracking-file-read.db #1 DB.locking false #2 read_from_head true #3 tag tracking-file-read [OUTPUT] name stdout match *
Inspect the complete tail configuration showing `DB`, `DB.locking`, `read_from_head`, tagging, and stdout output together. section 3.5.2
SQLite-backed progress tracking matters when processes start and stop, including environments where Kubernetes may evict Pods. It complements log-capture choices such as files, HTTP, forwarding, stdin, and application log appenders by preserving state specifically for long-running file readers. section 3.5.2 section 3.9
