How to list all available Kafka brokers in a cluster?

In a Kafka cluster, listing all available Kafka brokers is an important operation for monitoring and managing the health of the cluster. To retrieve a list of all available Kafka brokers in the cluster, several methods can be employed, such as using the zookeeper-shell command, the kafka-topics.sh script, or programmatically utilizing Kafka's Admin API. Below I will detail these methods:

1. Using Zookeeper-shell

Kafka uses Zookeeper to manage cluster metadata, including broker details. By connecting to the Zookeeper server, we can inspect the broker information stored within it. Here are the specific steps:

bash
# Connect to the Zookeeper server
zookeeper-shell.sh zookeeper-host:port

# View the list of all brokers
ls /brokers/ids

This will return a list of broker IDs. To retrieve detailed information for each broker, use the following command:

bash
get /brokers/ids/[broker_id]

Here, [broker_id] is one of the IDs returned by the previous command.

2. Using Kafka-topics.sh Script

Kafka includes several useful scripts, such as kafka-topics.sh, which can be used to view details of a topic and indirectly display broker information. For example:

bash
# List detailed information for a topic, including its associated brokers
kafka-topics.sh --describe --topic your-topic-name --bootstrap-server broker-host:port

Although this method requires specifying a topic name and does not directly return a list of all brokers, it provides a view of the relationship between brokers and topics.

3. Using Kafka Admin API

For scenarios requiring programmatic access to broker information, Kafka's Admin API can be utilized. Here is an example implementation in Java:

java
import org.apache.kafka.clients.admin.AdminClient;
import org.apache.kafka.clients.admin.AdminClientConfig;

import java.util.Properties;
import java.util.concurrent.ExecutionException;

public class BrokerListExample {
    public static void main(String[] args) {
        Properties config = new Properties();
        config.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, "broker-host:port");

        AdminClient admin = AdminClient.create(config);
        try {
            System.out.println("Brokers in the cluster: " + admin.describeCluster().nodes().get());
        } catch (InterruptedException | ExecutionException e) {
            e.printStackTrace();
        } finally {
            admin.close();
        }
    }
}

This code creates an AdminClient object and uses the describeCluster() method to retrieve cluster information, which includes a list of all active brokers.

Summary

By employing the methods above, we can effectively list all available brokers in a Kafka cluster. Different methods suit various use cases; for instance, Zookeeper commands can be used in maintenance scripts, while the Admin API is suitable for applications requiring dynamic information retrieval.