What are Prometheus Remote Write and Remote Read?

Prometheus Remote Write and Remote Read mechanisms:

Remote Write: Send data from Prometheus to remote storage systems.

Configuration Example:

yaml
remote_write:
  - url: "http://remote-storage:9201/api/v1/write"
    basic_auth:
      username: "user"
      password: "pass"
    queue_config:
      capacity: 10000
      max_shards: 50
      min_shards: 1
      max_samples_per_send: 1000
      batch_send_deadline: 5s
      min_backoff: 30ms
      max_backoff: 100ms
    write_relabel_configs:
      - source_labels: [__name__]
        regex: 'expensive_.*'
        action: drop

Use Cases:

• Long-term data storage
• Cross-cluster data aggregation
• Data analysis and reporting
• Backup and disaster recovery

Remote Read: Read data from remote storage systems.

Configuration Example:

yaml
remote_read:
  - url: "http://remote-storage:9201/api/v1/read"
    read_recent: true
    basic_auth:
      username: "user"
      password: "pass"

Use Cases:

• Query historical data
• Cross-datasource queries
• Data analysis

Supported Remote Storage:

• Thanos
• Cortex
• VictoriaMetrics
• InfluxDB
• M3DB
• TimescaleDB

Queue Configuration Parameters:

• capacity: Queue capacity
• max_shards: Maximum number of shards
• min_shards: Minimum number of shards
• max_samples_per_send: Maximum samples per send
• batch_send_deadline: Batch send timeout
• min_backoff / max_backoff: Backoff time

Best Practices:

1. Use write_relabel_configs to filter unnecessary data
2. Configure queue parameters reasonably to avoid memory overflow
3. Monitor Remote Write performance metrics
4. Use read_recent: true to improve query performance
5. Consider data compression to reduce network transmission

Monitoring Metrics:

• prometheus_remote_storage_queue_length
• prometheus_remote_storage_failed_samples_total
• prometheus_remote_storage_succeeded_samples_total