What are the advanced features of Zookeeper? How to use Watcher, ACL, and transaction operations?

Answer

Zookeeper provides multiple advanced features that make it more flexible and powerful in distributed systems.

1. Watcher Mechanism

Watcher Characteristics:

• One-time trigger: Automatically deleted after triggering
• Lightweight: Only notifies event type, does not include data
• Asynchronous notification: Processed through callback functions

Watcher Types:

java
// Node data changes
zk.getData("/path", watcher, null);

// Child node changes
zk.getChildren("/path", watcher);

// Node existence changes
zk.exists("/path", watcher);

Event Types:

• NodeCreated: Node created
• NodeDeleted: Node deleted
• NodeDataChanged: Node data changed
• NodeChildrenChanged: Child nodes changed

Best Practices:

• Re-register Watcher after it triggers
• Avoid time-consuming operations in Watcher
• Use exists() to monitor non-existent nodes

2. ACL Permission Control

Permission Types:

• CREATE: Create child nodes
• READ: Read node data
• WRITE: Update node data
• DELETE: Delete child nodes
• ADMIN: Set ACL

Permission Schemes:

java
// world: anyone
ZooDefs.Ids.OPEN_ACL_UNSAFE

// auth: authenticated user
new ACL(Perms.ALL, new Id("auth", "username:password"))

// digest: username password
new ACL(Perms.READ, new Id("digest", "username:password"))

// ip: IP address
new ACL(Perms.READ, new Id("ip", "192.168.1.1"))

// super: super administrator

Setting ACL:

java
// Set ACL when creating node
zk.create("/secure", data, 
    ZooDefs.Ids.READ_ACL_UNSAFE, 
    CreateMode.PERSISTENT);

// Modify node ACL
zk.setACL("/secure", 
    ZooDefs.Ids.OPEN_ACL_UNSAFE, 
    -1);

3. Transaction Operations

Transaction Characteristics:

• Atomicity: Either all succeed or all fail
• Sequentiality: Execute in submission order

multi Operation:

java
List<Op> ops = new ArrayList<>();

// Create node
ops.add(Op.create("/multi/node1", 
    "data1".getBytes(), 
    ZooDefs.Ids.OPEN_ACL_UNSAFE, 
    CreateMode.PERSISTENT));

// Update data
ops.add(Op.setData("/multi/node1", 
    "newData".getBytes(), 
    -1));

// Delete node
ops.add(Op.delete("/multi/node1", -1));

// Execute transaction
zk.multi(ops);

4. Four-Letter Commands

Common Four-Letter Commands:

bash
# View cluster status
echo stat | nc localhost 2181

# View connection information
echo cons | nc localhost 2181

# View environment variables
echo envi | nc localhost 2181

# View configuration
echo conf | nc localhost 2181

# View monitoring information
echo mntr | nc localhost 2181

# View node statistics
echo dump | nc localhost 2181

# Reset connection statistics
echo srst | nc localhost 2181

# View server status
echo srvr | nc localhost 2181

# View watcher information
echo wchs | nc localhost 2181

5. Data Snapshots and Transaction Logs

Transaction Logs:

• Record all write operations
• Used for data recovery
• Sequential write, high performance

Snapshots:

• Periodically save memory state
• Accelerate startup recovery
• Compressed storage

Recovery Process:

1. Load latest snapshot
2. Apply transaction logs after snapshot
3. Sync differential data with Leader

6. Client Reconnection Mechanism

Automatic Reconnection:

java
// Set retry policy
RetryPolicy retryPolicy = new ExponentialBackoffRetry(
    1000,  // base sleep time
    3      // max retries
);

CuratorFramework client = CuratorFrameworkFactory.builder()
    .connectString("localhost:2181")
    .retryPolicy(retryPolicy)
    .build();

Retry Policies:

• ExponentialBackoffRetry: Exponential backoff
• RetryNTimes: Fixed number of retries
• RetryUntilElapsed: Timeout retry
• RetryOneTime: Single retry

7. Session Management

Session States:

• CONNECTING: Connecting
• CONNECTED: Connected
• RECONNECTING: Reconnecting
• CLOSED: Closed

Session Timeout:

• Client heartbeat maintains session
• Ephemeral nodes automatically deleted after timeout
• Configurable timeout

Session Recovery:

java
// Recover using session ID and password
byte[] password = zk.getSessionPasswd();
long sessionId = zk.getSessionId();

ZooKeeper newZk = new ZooKeeper(
    "localhost:2181",
    30000,
    watcher,
    sessionId,
    password
);

8. Container Nodes (3.5+)

Container Node Characteristics:

• Automatically deleted when no child nodes
• Used for dynamic resource management

Use Cases:

• Parent node for locks
• Temporary resource groups

java
// Create container node
zk.create("/container", 
    null, 
    ZooDefs.Ids.OPEN_ACL_UNSAFE, 
    CreateMode.CONTAINER);

9. TTL Nodes (3.5+)

TTL Node Characteristics:

• Set expiration time
• Automatically deleted when expired
• Need to enable TTL feature

Enable TTL:

properties
# zoo.cfg
zookeeper.extendedTypesEnabled=true

Create TTL Node:

java
// Create TTL node
zk.create("/ttl-node", 
    data, 
    ZooDefs.Ids.OPEN_ACL_UNSAFE, 
    CreateMode.PERSISTENT_WITH_TTL, 
    new Stat(), 
    5000);  // TTL 5 seconds

10. Advanced Client Curator

Curator Framework Features:

• Connection management
• Retry mechanism
• Distributed lock
• Leader election
• Distributed counter
• Distributed queue

Distributed Lock Example:

java
InterProcessMutex lock = new InterProcessMutex(
    client, 
    "/locks/my-lock"
);

try {
    // Acquire lock
    lock.acquire();
    
    // Execute business logic
    doSomething();
    
} finally {
    // Release lock
    lock.release();
}

Leader Election Example:

java
LeaderSelectorListener listener = new LeaderSelectorListener() {
    @Override
    public void takeLeadership() {
        // Execute after becoming Leader
        while (true) {
            // Maintain Leader status
            Thread.sleep(1000);
        }
    }
};

LeaderSelector selector = new LeaderSelector(
    client, 
    "/leader", 
    listener
);
selector.start();

11. Data Migration and Backup

Data Export:

bash
# Export data using zkCli
zkCli.sh -server localhost:2181
get /path > backup.txt

Data Import:

bash
# Import data
zkCli.sh -server localhost:2181
create /path "data"

Cluster Migration:

1. Stop writes
2. Export data
3. Import to new cluster
4. Switch client connections

12. Monitoring and Alerting

Monitoring Metrics:

• Node status
• Latency metrics
• Throughput
• Connection count
• Memory usage

Alerting Strategies:

• Leader switch alert
• Latency threshold alert
• Connection limit alert
• Memory usage alert

13. Security Hardening

Security Measures:

• Enable SASL authentication
• Configure ACL permissions
• Network isolation
• Regular backups
• Log auditing

SASL Authentication Configuration:

properties
# jaas.conf
Server {
    org.apache.zookeeper.server.auth.DigestLoginModule required
    user_super="admin";
};

Client {
    org.apache.zookeeper.server.auth.DigestLoginModule required
    username="admin"
    password="admin";
};