The prometheus_remote_write plugin is a Fluent Bit output that pushes metrics to endpoints implementing Prometheus remote write. It provides an alternative to the scrape model: Fluent Bit can act as the local metrics agent, collect metrics through inputs such as fluentbit_metrics, and send them onward rather than waiting for Prometheus to poll an exporter. section 5.6 section 5.6.4
Push delivery is useful when scraping may miss short-lived metric producers. A Pod can disappear before Prometheus discovers and polls its exporter; serverless workloads such as Fn Project and AWS Lambda have the same risk. Remote write also avoids exposing node exporters for cross-network scraping, which can otherwise require proxies, API gateways, and added authentication controls. Sending data when it is produced can avoid polling endpoints that have no new metrics, while reducing the delay before critical thresholds become visible. section 5.6.4
Configure the plugin as an output by choosing the destination and transport details:
- Set name prometheus_remote_write and use match to select the metric records to forward. section 5.6.4
- Set host, uri, and port for the remote-write endpoint; the examples use /api/prom/push for a hosted Grafana endpoint and /api/v1/write for a local Prometheus endpoint. section 5.6.4
- Enable TLS for the HTTPS hosted endpoint or disable it for the illustrated local HTTP endpoint, according to the destination. section 5.6.4
- Provide http_user and http_passwd when the destination requires credentials, and enable log_response_payload when response diagnostics are needed. section 5.6.4
- Use add_label to distinguish metric origins, such as fluentbit=cloud and fluentbit=local. section 5.6.4
Forward Fluent Bit internal metrics to two destinations
Use fluentbit_metrics with tag internal_metrics and a five-second scrape interval, then match that tag in two prometheus_remote_write outputs. One output can send labeled cloud metrics over TLS to a hosted endpoint on port 443 with HTTP credentials; another can send labeled local metrics to a Prometheus endpoint on port 9090 without TLS. Start the configuration with fluent-bit -c fluentbit/prometheus-remote-output.conf. section 5.6.4
Listing 5.8 Prometheus writer: prometheus-remote-output.conf
[SERVICE]
flush 5
log_level trace
[INPUT]
name fluentbit_metrics
tag internal_metrics
scrape_interval 5
[OUTPUT]
name prometheus_remote_write #1
match *
host prometheus-prod-05-gb-south-0.grafana.net #2
uri /api/prom/push #3
port 443
tls on
log_response_payload on
add_label fluentbit cloud
http_user ${my-userid} #4
http_passwd ${my-password}
[OUTPUT]
name prometheus_remote_write
match *
host 192.168.0.205 #5
uri /api/v1/write #6
port 9090 #7
tls off
log_response_payload on #8
add_label fluentbit local #9
Inspect the complete configuration to compare hosted and local remote-write outputs, including endpoint paths, TLS settings, credentials, labels, and response logging. section 5.6.4
In a distributed enterprise deployment, an egress Fluent Bit node can send its local metrics to a central node with prometheus_remote_write. The number of reporting remote nodes can indicate lost contact with a managed or SaaS instance. Central operations can visualize received metrics in Prometheus and Grafana and alert on upper or lower thresholds. section 11.7
