How to integrate Prometheus with Grafana and what are the best practices?

Integration and best practices of Prometheus with Grafana:

Integration Configuration:

1. Add Prometheus Data Source:

json
{
  "name": "Prometheus",
  "type": "prometheus",
  "url": "http://prometheus:9090",
  "access": "proxy",
  "isDefault": true
}

1. Create Dashboards:

• Use variables for dynamic queries
• Use template variables for multi-environment switching
• Configure alert panels

Common Query Examples:

1. CPU Usage:

promql
100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

1. Memory Usage:

promql
(1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100

1. Disk Usage:

promql
(1 - (node_filesystem_avail_bytes{fstype!="tmpfs"} / node_filesystem_size_bytes)) * 100

1. Network Traffic:

promql
rate(container_network_receive_bytes_total[5m])

Variable Configuration Example:

yaml
# Instance variable
instance: label_values(up, instance)

# Namespace variable
namespace: label_values(kube_pod_info, namespace)

# Time range variable
interval: 30s, 1m, 5m, 15m, 1h

Alert Configuration:

• Configure alert rules in Grafana
• Support multiple notification channels (email, Slack, Webhook)
• Can integrate with Prometheus Alertmanager

Best Practices:

1. Dashboard Organization:

   • Categorize by business or system
   • Use folders for management
   • Add descriptions and tags

2. Query Optimization:

   • Use Recording Rules for pre-computation
   • Avoid complex queries
   • Set reasonable refresh intervals

3. Visualization Tips:

   • Choose appropriate chart types
   • Use threshold annotations
   • Add legends and annotations

4. Permission Management:

   • Configure role-based access control
   • Restrict access to sensitive data
   • Use API Keys for automation

Import Community Dashboards:

• Use Grafana official dashboard library
• Search keywords: Prometheus, Kubernetes, Node Exporter
• Customize based on requirements