What is the format of logstash config file

Logstash configuration files primarily consist of three sections: input, filter, and output. Each section defines a distinct stage in the Logstash data processing pipeline. Configuration files are typically written in Logstash's custom language, which is based on Apache Groovy. Here is a simple example illustrating how these sections function:

1. Input Section

The input section specifies how Logstash receives data. For example, data can be sourced from files, specific ports, or particular services.

ruby
input {
  file {
    path => "/path/to/your/logfile.log"
    start_position => "beginning"
  }
}

In this example, Logstash is configured to read data from the specified file path, where start_position => "beginning" indicates reading from the start of the file.

2. Filter Section

The filter section processes data before it is sent to the output. For instance, you can parse, modify, or transform data here.

ruby
filter {
  grok {
    match => { "message" => "%{COMBINEDAPACHELOG}" }
  }
}

Here, the grok plugin parses standard Apache log files, breaking them into a format that is easily understandable and queryable.

3. Output Section

The output section defines where data is sent. Data can be output to files, terminals, databases, or other Logstash instances.

ruby
output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "logstash-%{+YYYY.MM.dd}"
  }
  stdout { codec => rubydebug }
}

In this configuration, processed data is sent to the Elasticsearch service with a new index created daily. Additionally, data is output to the console for viewing during development or debugging.

These three sections collaborate to form a robust data processing pipeline, capable of receiving data from multiple sources, processing it as required, and outputting it to one or more destinations. The entire configuration file is typically saved as a .conf file, such as logstash.conf.