What are common Zookeeper problems? How to solve connection timeout, split-brain, and data inconsistency issues? - 面试题

Answer

In the process of using Zookeeper, various problems are often encountered. Understanding these problems and their solutions is crucial for both operations and development.

1. Connection Timeout Issue

Problem Description: Clients frequently experience connection timeouts when connecting to Zookeeper.

Possible Causes:

High network latency
Session Timeout set too short
High server load
Firewall blocking connections

Solutions:

java
// Increase Session Timeout
ZooKeeper zk = new ZooKeeper(
    "localhost:2181",
    30000,  // 30 seconds
    watcher
);

// Check network connectivity
ping zookeeper-server

// Check firewall
telnet zookeeper-server 2181

// Monitor server load
echo mntr | nc localhost 2181

2. Split-Brain Issue

Problem Description: Multiple Leaders appear in the cluster, causing data inconsistency.

Possible Causes:

Network partition
Even number of nodes
Incorrect election algorithm configuration

Solutions:

Use odd number of nodes (3, 5, 7)
Configure reasonable election timeout
Monitor cluster status
Use Zookeeper's majority mechanism to avoid split-brain

properties
# Configure election timeout
electionTimeout=3000

3. Data Inconsistency Issue

Problem Description: Different nodes read inconsistent data.

Possible Causes:

Reading stale data
Follower sync delay
Network partition

Solutions:

java
// Use sync() to force synchronization
zk.sync("/path", (rc, path, ctx) -> {
    // Read after sync completes
    zk.getData("/path", false, stat);
}, null);

// Monitor sync delay
echo mntr | nc localhost 2181 | grep -E "zk_synced"

4. Memory Overflow Issue

Problem Description: Zookeeper server memory overflow, causing service unavailability.

Possible Causes:

Too many node data
Too many Watchers
Too many client connections
JVM heap memory set too small

Solutions:

bash
# Increase JVM heap memory
-Xms4g -Xmx4g

# Use G1 GC
-XX:+UseG1GC
-XX:MaxGCPauseMillis=200

# Monitor memory usage
jmap -heap <pid>

# Clean up unused nodes
deleteall /old/path

5. Poor Write Performance Issue

Problem Description: High write operation latency, low throughput.

Possible Causes:

High Leader load
High network latency
Slow disk I/O
Unoptimized transaction logs

Solutions:

properties
# Separate transaction logs and data snapshots
dataLogDir=/data/zookeeper/logs
dataDir=/data/zookeeper/data

# Use SSD for transaction logs

# Increase snapshot interval
snapCount=100000

# Optimize network
# Use low-latency network

6. Watcher Leak Issue

Problem Description: Watcher count continues to grow, causing memory leak.

Possible Causes:

Watchers not properly cleaned up
Repeated Watcher registration
Exceptions causing Watchers not deleted

Solutions:

java
// Use one-time Watcher
zk.getData("/path", event -> {
    // Handle event
    handleEvent(event);
    
    // Re-register
    zk.getData("/path", this, null);
}, null);

// Monitor Watcher count
echo wchs | nc localhost 2181

// Regularly clean up unused Watchers

7. Frequent Election Issue

Problem Description: Cluster frequently performs Leader election, affecting service availability.

Possible Causes:

Unstable network
Insufficient node resources
Election timeout set too short
High Leader load

Solutions:

properties
# Increase election timeout
electionTimeout=5000

# Optimize network
# Increase node resources

# Monitor election count
echo stat | nc localhost 2181 | grep -E "Mode"

8. Large Node Data Issue

Problem Description: Single node data exceeds 1MB, causing performance degradation.

Possible Causes:

Unreasonable design
Data not sharded
Large file storage

Solutions:

java
// Shard data storage
for (int i = 0; i < chunks; i++) {
    String path = "/data/chunk-" + i;
    byte[] chunk = data[i];
    zk.create(path, chunk, 
        ZooDefs.Ids.OPEN_ACL_UNSAFE, 
        CreateMode.PERSISTENT);
}

// Use external storage for large files
// Zookeeper only stores file path

9. Client Connection Leak Issue

Problem Description: Client connection count continues to grow, reaching limit.

Possible Causes:

Connections not properly closed
Improper connection pool configuration
Exceptions causing connections not released

Solutions:

java
// Use try-with-resources
try (ZooKeeper zk = new ZooKeeper(...)) {
    // Use zk
}

// Use connection pool
CuratorFramework client = CuratorFrameworkFactory.builder()
    .connectString("localhost:2181")
    .retryPolicy(new ExponentialBackoffRetry(1000, 3))
    .build();

// Monitor connection count
echo cons | nc localhost 2181

10. Cluster Scaling Issue

Problem Description: Slow data synchronization during cluster scaling, affecting service.

Possible Causes:

Large data on new node
Insufficient network bandwidth
Unoptimized sync mechanism

Solutions:

bash
# 1. Add all servers in new node config file
# 2. Start new node
# 3. Wait for data sync to complete
# 4. Monitor sync status

# Monitor sync status
echo stat | nc localhost 2181 | grep -E "Mode|Zxid"

# Use snapshot to accelerate sync
# Copy snapshot file from existing node

11. Permission Issue

Problem Description: Client cannot access node, prompted with insufficient permissions.

Possible Causes:

Incorrect ACL configuration
Authentication failure
Improper permission settings

Solutions:

java
// Check node ACL
List<ACL> acls = zk.getACL("/path", stat);

// Modify node ACL
zk.setACL("/path", 
    ZooDefs.Ids.OPEN_ACL_UNSAFE, 
    -1);

// Add authentication info
zk.addAuthInfo("digest", "username:password".getBytes());

12. Version Compatibility Issue

Problem Description: Abnormal communication between different version Zookeeper clusters.

Possible Causes:

Too large version difference
Incompatible protocols
Unsupported features

Solutions:

bash
# Check version
echo stat | nc localhost 2181 | grep -E "Zookeeper version"

# Upgrade version (rolling upgrade)
# 1. Upgrade Follower
# 2. Upgrade Leader
# 3. Verify cluster status

# Maintain version consistency
# Use same version of Zookeeper

13. Monitoring and Alerting Issue

Problem Description: Unable to detect cluster anomalies in time.

Possible Causes:

Improper monitoring configuration
Unreasonable alert thresholds
Incomplete monitoring metrics

Solutions:

bash
# Key monitoring metrics
# 1. Latency metrics
echo mntr | nc localhost 2181 | grep -E "latency"

# 2. Throughput metrics
echo mntr | nc localhost 2181 | grep -E "packets"

# 3. Connection count metrics
echo cons | nc localhost 2181 | wc -l

# 4. Memory usage
jmap -heap <pid>

# Configure alerts
# Alert when latency > 10ms
# Alert when connection count > 1000
# Alert when memory usage > 80%

14. Data Recovery Issue

Problem Description: Data loss or unable to recover after cluster failure.

Possible Causes:

Corrupted transaction logs
Lost snapshot files
Improper backup strategy

Solutions:

bash
# Regular backup
# 1. Backup transaction logs
cp -r /data/zookeeper/logs /backup/

# 2. Backup snapshot files
cp -r /data/zookeeper/data /backup/

# Data recovery
# 1. Stop cluster
# 2. Restore backup files
# 3. Start cluster
# 4. Verify data integrity

# Use snapshot and transaction logs for recovery
zkServer.sh start

15. Performance Bottleneck Issue

Problem Description: Cluster performance cannot meet business requirements.

Possible Causes:

Unreasonable architecture design
Insufficient resource configuration
Improper data model design

Solutions:

properties
# Add Observer nodes to improve read performance

# Optimize data model
# Reduce node hierarchy
# Control node data size
# Properly use ephemeral nodes

# Increase cluster scale
# Expand from 3 nodes to 5 nodes

# Optimize configuration parameters
tickTime=2000
maxClientCnxns=100

Preventive Measures

Regular Monitoring: Establish comprehensive monitoring system
Capacity Planning: Plan resource requirements in advance
Backup Strategy: Regularly backup data
Documentation: Record configurations and changes
Drill Testing: Regularly conduct fault drills
Version Management: Unified version management
Security Hardening: Configure ACL and authentication