What is the basic structure of a Logstash configuration file, and how do you write a complete configuration?

The basic structure of a Logstash configuration file contains three main parts: input, filter, and output. Each part can contain multiple plugin configurations.

Configuration File Structure

conf
input {
  # Input plugin configuration
}

filter {
  # Filter plugin configuration
}

output {
  # Output plugin configuration
}

1. Input Configuration

The Input section defines data sources, common configuration examples:

File Input

conf
input {
  file {
    path => "/var/log/*.log"
    start_position => "beginning"
    sincedb_path => "/dev/null"
    type => "syslog"
  }
}

Beats Input

conf
input {
  beats {
    port => 5044
  }
}

Kafka Input

conf
input {
  kafka {
    bootstrap_servers => "localhost:9092"
    topics => ["logs"]
    group_id => "logstash-consumer"
  }
}

2. Filter Configuration

The Filter section processes and transforms data, commonly used filters:

Grok Filter

conf
filter {
  grok {
    match => { "message" => "%{COMBINEDAPACHELOG}" }
  }
}

Date Filter

conf
filter {
  date {
    match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"]
  }
}

Mutate Filter

conf
filter {
  mutate {
    rename => { "old_field" => "new_field" }
    remove_field => ["temp_field"]
    convert => { "status" => "integer" }
  }
}

GeoIP Filter

conf
filter {
  geoip {
    source => "client_ip"
    target => "geoip"
  }
}

3. Output Configuration

The Output section defines data output targets:

Elasticsearch Output

conf
output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "logstash-%{+YYYY.MM.dd}"
    document_type => "_doc"
  }
}

File Output

conf
output {
  file {
    path => "/path/to/output.log"
  }
}

Standard Output

conf
output {
  stdout {
    codec => rubydebug
  }
}

Conditional Statements

Logstash supports conditional statements to control data flow:

conf
filter {
  if [type] == "apache" {
    grok {
      match => { "message" => "%{COMBINEDAPACHELOG}" }
    }
  } else if [type] == "nginx" {
    grok {
      match => { "message" => "%{NGINXACCESS}" }
    }
  }
}

output {
  if [status] >= 400 {
    elasticsearch {
      hosts => ["http://localhost:9200"]
      index => "error-logs-%{+YYYY.MM.dd}"
    }
  } else {
    elasticsearch {
      hosts => ["http://localhost:9200"]
      index => "access-logs-%{+YYYY.MM.dd}"
    }
  }
}

Configuration File Validation

Use the following command to validate configuration file syntax:

bash
bin/logstash --config.test_and_exit -f /path/to/config.conf

Best Practices

1. Modular Configuration: Split different functional configurations into multiple files
2. Use Conditional Statements: Apply different processing logic based on data types
3. Reasonable Use of Filters: Avoid unnecessary filters to improve performance
4. Log Level Settings: Use appropriate log levels in production environments
5. Configuration File Management: Use version control systems to manage configuration files