liveBook · Manning

Connection pooling

As explained in Logs and Telemetry

Connection pooling is the practice of maintaining reusable database connections so operations can execute without repeatedly establishing new connections.

Connection pooling keeps multiple PostgreSQL connections available so Fluent Bit can write log events concurrently instead of paying the startup cost for each additional connection. Enable asynchronous output with async true, then set min_pool_size for maintained connections and max_pool_size for the concurrency limit. section 5.7

Pooling changes database output behavior in practical ways:

  • Each connection can process a chunk of log events, increasing potential write throughput. section 5.7
  • Concurrent servicing can store events slightly out of sequence; use SQL ORDER BY clauses when reading them. section 5.7
  • Test under load with LogSimulator, while recognizing that CPU, memory, Docker, and simulator settings affect whether extra connections are used. section 5.7
Listing 5.9 PostgreSQL output example: db-output.conf
[SERVICE]
  flush 1
  log_level info

[INPUT]
  name http    
  listen 0.0.0.0
  port 9881
  tag input1

[OUTPUT]
  name  pgsql       #1
  match  *
  host  localhost      #2
  port  5455
  user  postgresUser      #3
  password  postgresPW
  database  postgres      #4
  table   fluentbit
  timestamp_key ts        #5

Use the PostgreSQL output configuration as the base before adding asynchronous pooling attributes. section 5.7

How Logs and Telemetry covers it

How deep it goes
Explains async activation, minimum and maximum pool sizing, ordering consequences, and load testing without detailing pool internals.
Examples
Uses a Fluent Bit PostgreSQL output configuration, pgAdmin record retrieval, and LogSimulator load generation.
What it assumes
Assumes a working Fluent Bit-to-PostgreSQL output setup and SQL queries for retrieving records.
What you can do after
Operate/configure it by enabling async output and setting min_pool_size and max_pool_size for PostgreSQL writes.
How it connects
Connects PostgreSQL output to Fluent Bit, asynchronous processing, SQL ORDER BY, Docker, and LogSimulator.
Cover of Logs and Telemetry
liveBookmanning.com
Read Logs and Telemetry on liveBook Get the book at manning.com

Also explained in

Compare how 9 books explain connection pooling →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Logs and Telemetry. The section links point to the parts of the book it draws on.