How to relabel consul sd config to get service discovery stats based on consul metadata?

When using Consul for service discovery, we often need to retrieve specific metrics based on service metadata. The relabeling functionality allows us to handle this metadata more flexibly, enabling more precise service monitoring and management. I will explain how to achieve retrieving service discovery metrics based on Consul metadata through the following steps:

Step 1: Configure Consul Service

First, ensure that each service includes necessary metadata when registered with Consul. For example, when registering a service, we can add custom tags or metadata such as environment information (production, staging), version numbers, etc. This information typically appears in the Consul configuration file as:

json
{
  "service": {
    "name": "web-app",
    "tags": [
      "production",
      "version_1.2"
    ],
    "address": "10.0.0.1",
    "port": 80
  }
}

Step 2: Configure Prometheus or Other Monitoring Tools

For example, with Prometheus, it supports relabeling services obtained from Consul service discovery through the relabel_configs option in its configuration file. We can add, modify, or replace labels based on service metadata. This allows us to dynamically adjust query and reporting logic based on specific requirements.

Here is an example configuration for relabeling based on service tags:

yaml
scrape_configs:
  - job_name: 'consul-services'
    consul_sd_configs:
      - server: 'localhost:8500'
        services: []
    relabel_configs:
      - source_labels: ['__meta_consul_tags']
        regex: '.*,environment_(.+?),.*'
        target_label: 'environment'
        replacement: '$1'
      - source_labels: ['__meta_consul_tags']
        regex: '.*,version_(.+?),.*'
        target_label: 'version'
        replacement: '$1'

In this configuration, we use regular expressions to extract environment and version information from __meta_consul_tags and map them to new labels environment and version.

Step 3: Query and Visualization

Once our monitoring system starts scraping data based on the relabeled information, we can more easily query and visualize data based on dimensions such as environment and version. In tools like Grafana, we can create more targeted charts and alerts based on these new labels.

Example

Suppose we have two service instances, one in production and one in development. With the above configuration, we can query performance metrics for these environments in Prometheus, such as:

promql
rate(http_requests_total{environment="production"}[5m])
rate(http_requests_total{environment="development"}[5m])

Such a configuration makes monitoring data analysis and troubleshooting more efficient, as we can quickly pinpoint the specific environment and version where issues occur.

Conclusion

Through the above steps, we can leverage Consul's service discovery capabilities and Prometheus's powerful label transformation to monitor service status flexibly and effectively, ensuring our services run healthily and stably.