What are the commonly used plugins in Logstash, and how do you install and manage plugins? - 面试题

Logstash provides a rich plugin ecosystem that can extend functionality through plugins. Here is content related to Logstash plugins.

Plugin Types

Logstash plugins are mainly divided into three categories:

1. Input Plugins

Responsible for reading data from data sources.

Common Plugins:

file: Read files from the file system
beats: Receive data from Beats
kafka: Consume data from Kafka
http: Receive data through HTTP interface
tcp/udp: Receive TCP/UDP data
syslog: Receive system logs
jdbc: Read data from databases
redis: Read data from Redis
s3: Read data from AWS S3
elasticsearch: Read data from Elasticsearch

2. Filter Plugins

Responsible for processing and transforming data.

Common Plugins:

grok: Parse unstructured data
mutate: Field operations (rename, delete, convert, etc.)
date: Parse timestamps
geoip: Add geographic location information
useragent: Parse User-Agent
json: Parse JSON data
csv: Parse CSV data
ruby: Process data using Ruby code
aggregate: Aggregate multiple events
drop: Drop events

3. Output Plugins

Responsible for sending data to target systems.

Common Plugins:

elasticsearch: Send to Elasticsearch
file: Write to files
kafka: Send to Kafka
redis: Send to Redis
http: Send data through HTTP
stdout: Output to standard output
email: Send emails
s3: Send to AWS S3
mongodb: Send to MongoDB

Plugin Management

1. View Installed Plugins

bash
bin/logstash-plugin list

2. View Plugin Details

bash
bin/logstash-plugin list --verbose

3. Install Plugins

bash
# Install from official repository
bin/logstash-plugin install logstash-output-s3

# Install specific version
bin/logstash-plugin install logstash-output-s3 --version 10.0.0

# Install from local file
bin/logstash-plugin install /path/to/plugin.zip

4. Update Plugins

bash
# Update all plugins
bin/logstash-plugin update

# Update specific plugin
bin/logstash-plugin update logstash-output-s3

5. Uninstall Plugins

bash
bin/logstash-plugin uninstall logstash-output-s3

6. Verify Plugins

bash
bin/logstash-plugin verify

Common Plugins in Detail

1. File Input Plugin

conf
input {
  file {
    path => "/var/log/*.log"
    start_position => "beginning"
    sincedb_path => "/dev/null"
    type => "syslog"
    tags => ["system"]
  }
}

2. Grok Filter Plugin

conf
filter {
  grok {
    match => {
      "message" => "%{COMBINEDAPACHELOG}"
    }
    patterns_dir => ["/path/to/patterns"]
    overwrite => ["message"]
  }
}

3. Elasticsearch Output Plugin

conf
output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "logstash-%{+YYYY.MM.dd}"
    document_type => "_doc"
    flush_size => 500
    idle_flush_time => 1
  }
}

4. Kafka Output Plugin

conf
output {
  kafka {
    bootstrap_servers => "localhost:9092"
    topic_id => "logs"
    codec => "json"
    compression_type => "snappy"
  }
}

Custom Plugin Development

1. Choose Plugin Type

Choose to develop Input, Filter, or Output plugin based on requirements.

2. Create Plugin Project

bash
# Use Logstash plugin generator
gem install logstash-plugin-generator
logstash-plugin generate --type input --name myinput

3. Plugin Structure

shell
logstash-input-myinput/
├── lib/
│   └── logstash/
│       └── inputs/
│           └── myinput.rb
├── spec/
│   └── inputs/
│       └── myinput_spec.rb
├── Gemfile
├── logstash-input-myinput.gemspec
└── README.md

4. Plugin Code Example

ruby
# lib/logstash/inputs/myinput.rb
require "logstash/inputs/base"
require "logstash/namespace"
require "socket"

class LogStash::Inputs::Myinput < LogStash::Inputs::Base
  config_name "myinput"
  
  config :host, :validate => :string, :default => "0.0.0.0"
  config :port, :validate => :number, :required => true
  
  def register
    @logger.info("Registering myinput", :host => @host, :port => @port)
  end
  
  def run(queue)
    @server = TCPServer.new(@host, @port)
    loop do
      client = @server.accept
      Thread.new do
        begin
          while line = client.gets
            event = LogStash::Event.new("message" => line)
            decorate(event)
            queue << event
          end
        rescue => e
          @logger.error("Error", :exception => e)
        ensure
          client.close
        end
      end
    end
  end
  
  def stop
    @server.close if @server
  end
end

5. Build and Install Plugin

bash
# Build gem package
gem build logstash-input-myinput.gemspec

# Install plugin
bin/logstash-plugin install logstash-input-myinput-1.0.0.gem

Plugin Configuration Best Practices

1. Plugin Order

Arrange plugins in reasonable order according to data processing flow:

shell
Input → Filter → Output

2. Conditional Statements

Use conditional statements to avoid unnecessary plugin processing:

conf
filter {
  if [type] == "apache" {
    grok {
      match => { "message" => "%{COMBINEDAPACHELOG}" }
    }
  }
}

3. Error Handling

Handle plugin execution failures:

conf
filter {
  grok {
    match => { "message" => "%{PATTERN:field}" }
    tag_on_failure => ["_grokparsefailure"]
  }
  
  if "_grokparsefailure" in [tags] {
    # Handle parse failure
  }
}

4. Performance Optimization

Use batch processing to improve performance
Avoid using complex Ruby code
Reasonably configure thread count and batch size

Plugin Version Management

1. View Plugin Version

bash
bin/logstash-plugin list --verbose | grep logstash-output-s3

2. Lock Plugin Version

Specify plugin version in Gemfile:

ruby
gem "logstash-output-s3", "~> 10.0"

3. Version Compatibility

Ensure plugin version is compatible with Logstash version.

Plugin Testing

1. Unit Testing

ruby
# spec/inputs/myinput_spec.rb
require "logstash/devutils/rspec/spec_helper"
require "logstash/inputs/myinput"

describe LogStash::Inputs::Myinput do
  it "should register" do
    input = LogStash::Inputs::Myinput.new("port" => 1234)
    expect { input.register }.not_to raise_error
  end
end

2. Integration Testing

Use test data to verify plugin functionality.

Community Plugins

The Logstash community provides many third-party plugins, which can be found through:

Logstash official plugin repository
GitHub search
Elastic community forum

Best Practices

Choose Appropriate Plugins: Choose the most suitable plugin based on requirements
Keep Plugins Updated: Regularly update plugins to get latest features and fixes
Test Plugins: Thoroughly test plugins before using in production
Monitor Plugin Performance: Monitor performance metrics of plugins
Document: Document usage methods and configurations of custom plugins