Fluent Bit Multiline Parser: Java Stacktraces & Go Panic Logs

1. The Core Problem: Container Runtimes & Fragmented Exceptions

In cloud-native environments running Docker, containerd, or CRI-O, containerized applications write logs directly to standard output (stdout) and standard error (stderr). The underlying container runtime intercepts this raw byte stream and appends runtime metadata (timestamp, stream identifier, and partial-line tags), writing each physical line as an isolated record to disk:

2026-10-12T14:05:01.129381024Z stdout F 2026-10-12 14:05:01 ERROR [OrderController] NullPointerException: orderID is null
2026-10-12T14:05:01.129410123Z stdout F     at com.payment.service.OrderService.process(OrderService.java:142)
2026-10-12T14:05:01.129429401Z stdout F     at com.payment.controller.OrderController.submit(OrderController.java:58)
2026-10-12T14:05:01.129448291Z stdout F Caused by: java.lang.IllegalStateException: Database connection pool exhausted
2026-10-12T14:05:01.129462810Z stdout F     at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:214)

Without multiline stream reassembly, a simple 20-line stacktrace becomes 20 disconnected log entries inside Elasticsearch, Loki, or Datadog. Log correlation breaks, full-text searches fail to associate causes with callers, and alert thresholds trigger false alarms for every isolated line fragment.

2. Modern Fluent Bit Architecture: Input-Level vs Filter-Level Multiline

Historically, Fluent Bit users relied on the multiline filter plugin to assemble split lines. While functional, the filter approach required logs to be read from disk, parsed into internal message structures, serialized, routed to the filter stage, buffered in auxiliary memory, and re-emitted.

Starting in Fluent Bit v1.8 and perfected in v3.x, multiline processing moved natively into the [INPUT] plugin via the multiline.parser engine. This architectural shift provides three decisive advantages:

• Zero Inter-Filter Allocations: Multiline concatenation occurs directly in the file reader buffer before record creation, saving up to 60% memory overhead.
• Parser Cascading: Multiple parsers can be chained sequentially (e.g. first unwrapping CRI container headers, then parsing Java stacktraces).
• Predictable Flush Semantics: Buffer timeouts fire deterministically via non-blocking epoll timers, preventing indefinitely stalled logs.

3. Production Configuration: Tail Input with Built-In Parsers

Below is the production-grade fluent-bit.conf setup deployed across Kubernetes clusters for Spring Boot, Quarkus, and Go microservices:

[SERVICE]
    Flush         1
    Log_Level     info
    Daemon        off
    Parsers_File  parsers.conf
    Parsers_File  parsers_multiline.conf

[INPUT]
    Name              tail
    Tag               kube.*
    Path              /var/log/containers/*.log
    Parser            docker
    DB                /var/log/flb_kube.db
    Mem_Buf_Limit     50MB
    Skip_Long_Lines   On
    Refresh_Interval  10
    # Enable cascaded multiline parsers:
    # 1. cri: Strips containerd metadata (timestamp, stdout/stderr, F/P tag)
    # 2. java: Matches standard Java/JVM stacktrace patterns
    # 3. go: Matches Go runtime panic headers
    multiline.parser  cri, java, go
    multiline.flush_ms 500

[OUTPUT]
    Name            es
    Match           kube.*
    Host            elasticsearch-master.monitoring.svc
    Port            9200
    Index           app-logs-2026
    Type            _doc
    Trace_Error     On

The parameter multiline.flush_ms 500 instructs Fluent Bit to wait 500 milliseconds after seeing a candidate multiline entry. If no additional lines arrive within that window, the current buffer is finalized and flushed to prevent delivery latency on low-traffic microservices.

4. Custom Multiline Regex Parser Definition

When proprietary application frameworks emit non-standard logging formats, create a bespoke multiline state machine inside parsers_multiline.conf. A state machine parser defines an initial entry rule (start_state) and continuation transitions (cont):

[MULTILINE_PARSER]
    name          custom_enterprise_multiline
    type          regex
    flush_timeout 500

    # Rule 1: A new log entry begins with bracketed ISO-8601 timestamp and log level
    # Example: [2026-10-12 14:10:00.123] [ERROR] [worker-1]
    rule          "start_state"   "/^\[\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\.\d{3}\] \[[A-Z]+\]/"  "cont"

    # Rule 2: Stacktrace frames start with whitespace or 'at ' or 'Caused by:'
    rule          "cont"          "/^\s+(at |Caused by:|\.{3}|org\.|com\.|io\.)/"                 "cont"

    # Rule 3: Python tracebacks start with 'Traceback (most recent call last):'
    rule          "cont"          "/^(Traceback \(most recent call last\):|\s+File ".*", line)/"  "cont"

    # Rule 4: Go panics start with 'panic:' followed by goroutine dump
    rule          "cont"          "/^(panic: |goroutine \d+ \[running\]:|\s+created by )/"        "cont"

This unified regex state machine captures nested Java exceptions, Python tracebacks, and Go goroutine stack dumps without requiring separate ingestion paths.

5. Empirical Throughput & Memory Benchmark

We evaluated multiline aggregation performance across leading log shipping utilities on an enterprise Kubernetes node generating 100,000 log lines per second with a 15% exception stacktrace frequency:

6. Operational Pitfalls: Buffer Exhaustion & Truncation

When managing high-volume clusters, multiline parsers can cause memory pressure if application code enters recursive crash loops. Protect your cluster using these safeguards:

7. Frequently Asked Questions