A metric is a usually numeric measurement with a descriptive label that represents hardware or software state or activity. CPU use, memory, storage, network activity, and transactions per minute are typical examples. Alongside logs and traces, metrics are an observability signal used to monitor systems and understand operational behavior. section 1.1 section 3.6.7
Metrics make capacity and service behavior visible close to real time. Resource measurements can reveal whether services have sufficient CPU and memory, while business-oriented measurements can show transaction throughput. Collecting and aggregating these signals from distributed systems preserves evidence that may otherwise be lost during an incident and helps form a complete operational picture. section 1.1.1 section 3.2.1
A useful metric pipeline separates collection, identification, transformation, and delivery:
- Collect host measurements with inputs such as disk, cpu, mem, and netif. Configure a polling interval and explicit tags so the resulting events can be filtered or routed predictably. section 3.2.1
- Treat a metric as named measurement data. Simple metrics can be key-value pairs, but a large exporter output may require selecting only the names needed downstream. section 9.2.4
- Derive Prometheus metrics from JSON log events when an application emits monitoring values through logs. Map values to counters, gauges, or histograms, and add labels from log fields. section 7.5.2
- Filter metric volume with metrics_selector. Include or exclude literal names, prefixes, substrings, or regular-expression matches; prefer a string-prefix test over a regular expression when appropriate. section 9.2.4
- Route collected metrics with other telemetry to an aggregation or processing destination. Fluent Bit can participate in OpenTelemetry flows and connect to OTel-compatible services or analysis tools. section 1.3.2
Monitor a host and retain CPU metrics
Configure disk, cpu, mem, and netif inputs with a five-second interval and explicit tags, then send their output to stdout to inspect the records. When node exporter produces more data than required, apply metrics_selector with an include rule such as /^node_cpu*/ to retain CPU measurements only. Output can contain named values such as node_cpu_seconds_total with cpu and mode labels. section 3.2.1 section 9.2.4
Listing 3.1 Performing basic OS monitoring
[INPUT] name disk #1 interval_sec 5 tag book_vm_disk [INPUT] name cpu #2 interval_sec 5 tag book_vm_cpu [INPUT] name mem #3 interval_sec 5 tag book_vm_memory [INPUT] name netif #4 interval_sec 5 #interface eth0 #5 tag book_vm_network [OUTPUT] name stdout match * #6
Use the configuration to see four native host-monitoring inputs, their collection intervals, explicit tags, and a stdout output in one runnable pipeline. section 3.2.1
Listing 7.6 Generating Prometheus metrics: log-to-metrics.conf
[SERVICE]
flush 1
[INPUT]
name dummy
dummy {"greeting ":"hello", "gauge_value":"1"} #1
tag dummy.hello
[INPUT]
name dummy
dummy {" greeting":"bonjour", "gauge_value":"7"} #2
tag dummy.hello
[INPUT]
name dummy
dummy {"greeting":"goodbye", "gauge_value":"3"}
tag dummy.goodbye #3
[FILTER]
name log_to_metrics
match dummy.*
tag test_metric_all
metric_mode counter #4
label_field greeting #5
metric_name count_all_dummy_messages
metric_description count of messages by greeting
[FILTER]
name log_to_metrics #6
match dummy.*
tag test_metric_all
metric_mode counter
metric_name count_all_dummy_messages #7
metric_description count of ALL messages #8
[FILTER]
name log_to_metrics
match dummy.goodbye #9
tag goodbye_gauge
metric_mode gauge #10
metric_name goodbye_gauge
value_field gauge_value #11
metric_description A gauge from dummy.goodbye
[FILTER]
name log_to_metrics
match dummy.hello
tag hello_test_histogram
metric_mode histogram #12
metric_name hello_histogram
label_field greeting
bucket 0.4 #13
value_field gauge_value #14
metric_description A histogram for dummy.hello events
[OUTPUT]
name prometheus_exporter
match *
host 0.0.0.0
port 9885
Use the configuration to compare log-derived counters, gauges, and histograms, including metric names, descriptions, labels, numeric value fields, and histogram buckets. section 7.5.2
