How to optimize Zookeeper performance? What are the configuration parameters and architecture optimization recommendations?

Answer

Zookeeper performance optimization involves multiple levels, including configuration optimization, architecture design, and client optimization.

1. Configuration Parameter Optimization

Key Configuration Parameters:

properties
# Transaction log file size (recommended 64MB)
preAllocSize=65536

# Snapshot file size limit
snapCount=100000

# Client connection limit
maxClientCnxns=60

# Session timeout (adjust based on business)
tickTime=2000
initLimit=10
syncLimit=5

# Thread pool configuration
serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory

Optimization Recommendations:

• Set tickTime to 2000ms, avoid too short causing frequent timeouts
• Adjust maxClientCnxns based on actual connection count
• Use Netty instead of NIO to improve network performance

2. Storage Optimization

Transaction Log and Snapshot Separation:

properties
# Transaction log directory (high-performance disk)
dataLogDir=/data/zookeeper/logs

# Data snapshot directory (regular disk)
dataDir=/data/zookeeper/data

Optimization Strategies:

• Use SSD or high-performance disk for transaction logs
• Regular disks can be used for snapshots
• Regularly clean up old snapshot files

Auto-cleanup Configuration:

properties
# Number of snapshots to retain
autopurge.snapRetainCount=3

# Cleanup interval (hours)
autopurge.purgeInterval=1

3. Network Optimization

Network Configuration:

• Use low-latency network between nodes
• Avoid cross-datacenter deployment
• Increase network bandwidth

Connection Pool Optimization:

java
// Client connection pool configuration
ZooKeeper zk = new ZooKeeper(
    "host1:2181,host2:2181,host3:2181",
    30000,  // session timeout
    watcher,
    true    // canBeReadOnly
);

4. Cluster Architecture Optimization

Add Observer Nodes:

• Observer only handles read requests
• Does not participate in election and write voting
• Improves cluster read performance

Cluster Scale:

• 3 nodes: Suitable for small-scale applications
• 5 nodes: Recommended for production
• 7 nodes: Large-scale applications

Read-Write Separation:

• Write requests: Handled by Leader
• Read requests: Handled by Follower/Observer

5. Client Optimization

Connection Management:

• Use connection pool to reuse connections
• Set reasonable session timeout
• Implement reconnection mechanism

Watcher Optimization:

java
// Avoid registering Watcher repeatedly
zk.exists("/path", watcher);

// Use one-time Watcher
zk.getData("/path", event -> {
    // Re-register after handling event
    zk.getData("/path", this, null);
}, null);

Batch Operations:

• Use multi() to execute batch operations
• Reduce network round trips

6. Data Structure Optimization

Node Design Principles:

• Node hierarchy should not be too deep (recommended < 5 levels)
• Single node data size < 1MB
• Avoid frequent creation and deletion of nodes

Use Ephemeral Nodes:

• Ephemeral nodes are automatically cleaned up
• Reduce manual maintenance costs

Sequential Node Optimization:

• Use sequential nodes to implement queues
• Avoid large number of child nodes

7. Monitoring and Tuning

Key Monitoring Metrics:

1. Latency Metrics:

   • latency_avg: Average latency
   • latency_max: Maximum latency
   • Recommended target: < 10ms

2. Throughput Metrics:

   • packets_sent: Number of packets sent
   • packets_received: Number of packets received
   • Recommended target: > 10000 ops/s

3. Connection Metrics:

   • num_alive_connections: Number of active connections
   • Monitor connection leaks

4. Memory Metrics:

   • JVM heap memory usage
   • Recommended to keep below 70%

JVM Parameter Optimization:

bash
# Heap memory settings
-Xms2g -Xmx2g

# GC strategy
-XX:+UseG1GC
-XX:MaxGCPauseMillis=200

# GC logging
-Xloggc:/data/zookeeper/logs/gc.log
-XX:+PrintGCDetails

8. Common Performance Issues and Solutions

Issue 1: High Write Latency

• Cause: Network latency, slow disk I/O
• Solution: Optimize network, use SSD

Issue 2: Poor Read Performance

• Cause: Leader overload
• Solution: Add Observer nodes

Issue 3: Frequent Elections

• Cause: Network instability, insufficient node resources
• Solution: Optimize network, increase resources

Issue 4: Memory Overflow

• Cause: Too many nodes, Watcher leaks
• Solution: Clean up unused nodes, optimize Watchers

9. Performance Testing Recommendations

Testing Tools:

• zk-smoketest: Official testing tool
• Custom stress testing scripts

Testing Metrics:

• Throughput (ops/s)
• Latency (ms)
• Availability (%)

Testing Scenarios:

• Read-intensive
• Write-intensive
• Mixed

10. Best Practices

1. Plan cluster scale reasonably
2. Separate transaction logs and data snapshots
3. Use Observers to improve read performance
4. Optimize client connections and Watchers
5. Regular monitoring and tuning
6. Establish performance baselines
7. Good capacity planning