liveBook · Manning

Multiline parsing

As explained in Logs and Telemetry

Multiline parsing is the technique of determining event boundaries across multiple input lines and combining those lines into one event.

Multiline parsing reconstructs one logical log event from lines that arrive separately. It is useful for stack traces, formatted JSON, and other records containing newline characters. In Fluent Bit, parsing may happen at an input plugin or through a multiline filter; the parser identifies where an event starts, which lines continue it, and when it ends. section 6.6

Use the input-plugin approach when tailing container logs. A tail input can name one or more parsers with `multiline.parser`; for Kubernetes logs, `docker, cri` tries Docker format first and then CRI if parsing fails. Configure only the known format when the runtime is certain. section 4.7.6

Listing 4.9 Extract from Helm chart: values.yaml
luaScripts:                   #1
  filter_example.lua: |
    function filter_name(tag, timestamp, record)
        -- put your lua code here.
    end

config:                           #2
  service: |              #3
    [SERVICE]
        Daemon Off
        Flush {{ .Values.flush }}       #4
        Log_Level {{ .Values.logLevel }}
        Parsers_File /fluent-bit/etc/parsers.conf        #5
        Parsers_File /fluent-bit/etc/conf/custom_parsers.conf
        HTTP_Server On
        HTTP_Listen 0.0.0.0
        HTTP_Port {{ .Values.metricsPort }}
        Health_Check On

  inputs: |
    [INPUT]
        Name tail
        Path /var/log/containers/*.log
        multiline.parser docker, cri           #6
        Tag kube.*
        Mem_Buf_Limit 5MB       #7
        Skip_Long_Lines On

    [INPUT]
        Name systemd
        Tag host.*
        Systemd_Filter _SYSTEMD_UNIT=kubelet.service
        Read_From_Tail On

  filters: |
    [FILTER]
        Name kubernetes
        Match kube.*
        Merge_Log On
        Keep_Log Off
        K8S-Logging.Parser On
        K8S-Logging.Exclude On

    [FILTER]
        Name    lua
        Match   <your-tag>
        script  /fluent-bit/scripts/filter_example.lua  #8
        call    filter_name

  outputs: |
    [OUTPUT]
        Name es        #9
        Match kube.*
        Host elasticsearch-master
        Logstash_Format On
        Retry_Limit False

This Helm values fragment shows multiline parsing on a Kubernetes tail input, alongside container-log paths, tags, memory limits, and long-line handling. section 4.7.6

Use a multiline filter when records have already been ingested separately. Set `multiline.key_content` to the field containing log text and `multiline.parser` to a predefined or custom parser. With `buffer on`, Fluent Bit can consolidate same-tag records, but only when ordering and safeguards make accidental merging unlikely. section 6.6

Filter-based reconstruction

A tail input reads `multiline-json-small.txt` with the tag `multi-feed`. A multiline filter matches that tag, reads the `log` field, uses `multiline_Demo`, and enables buffering. Running `fluent-bit -c fluentbit/multiline-parser-filter.conf` emits the previously separate JSON lines as one `log` record.

Treat multiline parsing as a cost and capacity decision. Unlike single-line processing, it must determine the final line before proceeding. Bad termination rules can create large entries; use `emitter_storage.type`, `emitter_mem_buf_limit`, and `flush_ms` to control buffering and forced output. When possible, prevent multiline output upstream through logging practices or frameworks that remove newline characters. section 4.7.6 section 6.6

How Logs and Telemetry covers it

How deep it goes
Explains input and filter configuration, regex state rules, buffering, timeouts, storage, and failure risks.
Examples
Demonstrates Helm tail configuration and a runnable filter that joins multiline JSON into one output record.
What it assumes
Teaches custom parser rules from configuration examples while relying on Fluent Bit inputs, filters, and record fields.
What you can do after
Operate/configure Fluent Bit multiline parsing through tail inputs or filters while setting boundary, buffer, and flush controls.
How it connects
Connects tail inputs, parser filters, regex parsers, buffering, Kubernetes logs, Docker format, and CRI format.
Cover of Logs and Telemetry
liveBookmanning.com
Read Logs and Telemetry on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Logs and Telemetry. The section links point to the parts of the book it draws on.