What service discovery mechanisms does Prometheus support and how to configure them?

Detailed explanation of Prometheus service discovery mechanisms:

Static Configuration:

yaml
scrape_configs:
  - job_name: 'node'
    static_configs:
      - targets: ['192.168.1.10:9100', '192.168.1.11:9100']
        labels:
          datacenter: 'dc1'

Kubernetes Service Discovery:

yaml
scrape_configs:
  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_pod_ip]
        target_label: __address__
        replacement: $1:9104

Kubernetes Role Types:

• pod: Discover all Pods
• service: Discover all Services
• endpoints: Discover Service Endpoints
• endpointslices: Discover EndpointSlices
• node: Discover all Nodes
• ingress: Discover all Ingresses

Consul Service Discovery:

yaml
scrape_configs:
  - job_name: 'consul'
    consul_sd_configs:
      - server: 'consul.example.com:8500'
        services: ['web', 'api']
        tag_separator: ','
        scheme: http

DNS SRV Service Discovery:

yaml
scrape_configs:
  - job_name: 'dns'
    dns_sd_configs:
      - names: ['_prometheus._tcp.example.com']
        type: SRV
        port: 9100

EC2 Service Discovery:

yaml
scrape_configs:
  - job_name: 'ec2'
    ec2_sd_configs:
      - region: us-east-1
        access_key: AKIA...
        secret_key: xxx...

Relabel Configs:

• replace: Replace label values
• keep: Keep matching targets
• drop: Drop matching targets
• keep_if_equal: Keep if labels are equal
• drop_if_equal: Drop if labels are equal
• hashmod: Hash modulo
• labelmap: Label mapping

Common Metadata Labels:

• __meta_kubernetes_pod_name: Pod name
• __meta_kubernetes_service_name: Service name
• __meta_consul_service_metadata_*: Consul metadata
• __meta_ec2_tag_*: EC2 tags

Best Practices:

1. Use annotations to control Pod scraping
2. Use labels for grouping
3. Regularly clean up unused service discovery configs
4. Use relabel_configs to filter unwanted targets
5. Monitor service discovery performance impact