How to get topic list from kafka server in Java

Retrieving topic lists from Kafka servers in Java can be achieved using the Kafka AdminClient API. This API enables you to programmatically manage and inspect topics, including retrieving the list of existing topics. Below is a step-by-step guide on how to use AdminClient to retrieve topic lists from Kafka servers.

Step 1: Add Kafka client dependencies

First, ensure that your project includes the Kafka client library dependency. If you use Maven, add the following dependency to your pom.xml file:

xml
<dependency>
    <groupId>org.apache.kafka</groupId>
    <artifactId>kafka-clients</artifactId>
    <version>2.8.0</version> <!-- Use the version suitable for your project -->
</dependency>

Step 2: Configure and create AdminClient

Next, create an AdminClient instance by providing basic configurations, such as the Kafka server address (bootstrap.servers):

java
import org.apache.kafka.clients.admin.AdminClient;
import org.apache.kafka.clients.admin.AdminClientConfig;

import java.util.Properties;

Properties config = new Properties();
config.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092"); // Kafka server address

AdminClient adminClient = AdminClient.create(config);

Step 3: Retrieve topic lists

Using AdminClient, you can call the listTopics method to retrieve the list of topics:

java
import org.apache.kafka.clients.admin.ListTopicsOptions;
import org.apache.kafka.clients.admin.ListTopicsResult;
import java.util.Set;
import java.util.concurrent.ExecutionException;

try {
    ListTopicsOptions options = new ListTopicsOptions();
    options.listInternal(false); // set to false to exclude internal topics

    ListTopicsResult topics = adminClient.listTopics(options);
    Set<String> topicNames = topics.names().get(); // Retrieve the set of topic names
    for (String name : topicNames) {
        System.out.println(name);
    }
} catch (InterruptedException | ExecutionException e) {
    e.printStackTrace();
} finally {
    adminClient.close();
}

Example Explanation

In this example, we first set up the necessary configurations to connect to the Kafka server, then create an AdminClient instance. Using this instance, we call the listTopics() method to retrieve a set of all topic names and print them. Note that we use listInternal(false) to exclude topics used internally by Kafka.

Important Notes

• Ensure that the Kafka server address and port are configured correctly.
• Handle exceptions from asynchronous calls, such as InterruptedException and ExecutionException.
• Properly close AdminClient to release resources.

By following these steps, you can effectively retrieve all topic lists from the Kafka server within your Java application.