Polling collects data by repeatedly connecting to a service or endpoint and asking for the current state. In observability pipelines, Prometheus can scrape Fluent Bit metrics, while Fluent Bit can poll the Kubernetes API for cluster events. The interval controls how often requests occur, making polling a direct but recurring form of collection. section 5.6.4 section 4.7.3
Choose the interval as a trade-off between detection latency and request efficiency. Calls made too often can connect only to discover no new data; calls made too slowly can delay awareness of a metric reaching a critical threshold. Polling can also miss metrics from transient Pods, serverless workloads, or other processes that end before a collector reaches them. section 5.6.4
Account for operational constraints when polling Kubernetes events:
- Persist polling state in the plugin's SQLite database to avoid sending the same API events into the pipeline repeatedly. section 4.7.3
- Run only one active Fluent Bit instance for the events plugin; multiple instances retrieving the same events create duplicates. section 4.7.3
- Size that instance for cluster event volume, and rely on Kubernetes health monitoring rather than duplicate collectors. section 4.7.3
- Protect Kubernetes API credentials and certificates with Kubernetes secrets mounted into the Pod instead of environment variables. section 4.7.3
Polling Kubernetes events every five seconds
Configure the kubernetes_events input with kube_url, a service-account token file, and interval_sec 5. The plugin requests Kubernetes events on that schedule, tags them as kubernetes_events, and can send matching records to stdout for inspection. Request limits and retention settings help bound collection behavior. section 4.7.3
Listing 4.8 K8s events plugin illustration: kubernetes-out.conf
[SERVICE] flush 1 log_level debug [INPUT] name kubernetes_events #1 tag kubernetes_events kube_url https://kubernetes.default.svc #2 tls.verify off interval_sec 5 kube_token_file /var/run/secrets/kubernetes.io/serviceaccount/token #3 kube_request_limit 100 kube_retention_time 15m #kube_namespace #4 [OUTPUT] name stdout #5 match *
This configuration shows a Kubernetes events poller with its API URL, five-second interval, token-file path, request limit, retention time, and stdout output. section 4.7.3
Where direct scraping crosses networks or private subnets, exposing node exporters can require proxies, API gateways, and extra authentication. A push or streaming model avoids repeated reach-out calls and is attractive when data is short-lived, connectivity is constrained, or requests are likely to return nothing new. Fluent Bit's Prometheus remote-write output illustrates pushing metrics to remote or local Prometheus-compatible endpoints instead. section 5.6.4
Listing 5.8 Prometheus writer: prometheus-remote-output.conf
[SERVICE]
flush 5
log_level trace
[INPUT]
name fluentbit_metrics
tag internal_metrics
scrape_interval 5
[OUTPUT]
name prometheus_remote_write #1
match *
host prometheus-prod-05-gb-south-0.grafana.net #2
uri /api/prom/push #3
port 443
tls on
log_response_payload on
add_label fluentbit cloud
http_user ${my-userid} #4
http_passwd ${my-password}
[OUTPUT]
name prometheus_remote_write
match *
host 192.168.0.205 #5
uri /api/v1/write #6
port 9090 #7
tls off
log_response_payload on #8
add_label fluentbit local #9
This remote-write configuration contrasts polling by having Fluent Bit push metrics over TLS to a hosted endpoint or directly to a local Prometheus API. section 5.6.4
