How to process multiline log entry with logstash filter?

When using Logstash to process logs, handling multi-line log entries is a common yet complex challenge. Multi-line log entries commonly occur in stack traces, SQL queries, or other events that span multiple lines. To properly parse these log entries, utilize Logstash's multiline filter plugin.

Step 1: Identify the Log Entry Pattern

First, identify the starting pattern of log entries. For example, Java exception stack traces typically begin with a line containing the exception type and message, followed by multiple lines of stack information.

Step 2: Configure Logstash Input Plugin

In the Logstash configuration file, set up the input plugin to read log files. For instance, use the file plugin to read log files:

plaintext
input {
  file {
    path => "/path/to/your/logfile.log"
    start_position => "beginning"
  }
}

Step 3: Use the Multiline Filter

Next, use the multiline plugin to merge multi-line log entries. This is typically performed during the input phase to ensure log entries are complete before entering the filter. When configuring, specify when a line is considered a continuation of the previous line:

plaintext
codec => multiline {
  pattern => "^\s+" # Example: matching lines starting with whitespace
  what => "previous"
}

This configuration means that any line starting with whitespace is treated as a continuation of the previous line.

Step 4: Set Up Filters and Output

After configuring input and multiline processing, set up filters to refine log data as needed, and configure output, such as to Elasticsearch:

plaintext
output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "logdata"
  }
}

Example: Processing Java Exception Stack Traces

Suppose we have the following log format:

shell
Exception in thread "main" java.lang.NullPointerException
        at com.example.myapp.Main.main(Main.java:14)

We can configure multiline as follows:

plaintext
codec => multiline {
  pattern => "^\s+at"
  what => "previous"
}

This configuration merges lines starting with "at" into the previous line, as this is typical for Java stack traces.

By following these steps, Logstash can effectively process multi-line log entries, providing structured and complete data for subsequent log analysis.