How to change the number of replicas of a Kafka topic?

In Apache Kafka, changing the replication factor of a topic involves several key steps. Below, I will explain each step in detail and provide corresponding command examples.

Step 1: Review Existing Topic Configuration

First, we should review the current configuration of the topic, particularly the replication factor. This can be done using Kafka's kafka-topics.sh script. Assume the topic we want to modify is named my-topic; the following command can be used:

bash
bin/kafka-topics.sh --describe --topic my-topic --bootstrap-server localhost:9092

This command displays the current configuration of my-topic, including its replication factor.

Step 2: Prepare the JSON File for Reassignment

Changing the replication factor requires generating a reassignment plan in JSON format. This plan specifies how replicas of each partition should be distributed across different brokers. We can use the kafka-reassign-partitions.sh script to generate this file. Assume we want to increase the replication factor of my-topic to 3; the following command can be used:

bash
bin/kafka-reassign-partitions.sh --bootstrap-server localhost:9092 --topics-to-move-json-file topics.json --broker-list "1,2,3" --generate

The topics.json file should contain the topic information for modification, as shown below:

json
{
  "topics": [
    {"topic": "my-topic"}
  ],
  "version": 1
}

The broker-list specifies the brokers to which replicas should be assigned. This command outputs two JSON files: one for the current assignment and another for the proposed reassignment plan.

Step 3: Execute the Reassignment Plan

Once we have a satisfactory reassignment plan, we can apply it using the kafka-reassign-partitions.sh script:

bash
bin/kafka-reassign-partitions.sh --bootstrap-server localhost:9092 --reassignment-json-file expansion.json --execute

Here, expansion.json is the proposed reassignment plan generated in the previous step.

Step 4: Monitor the Reassignment Process

Reassigning replicas may take some time, depending on cluster size and load. We can monitor the status using the following command:

bash
bin/kafka-reassign-partitions.sh --bootstrap-server localhost:9092 --reassignment-json-file expansion.json --verify

This command informs us whether the reassignment was successful and the progress made.

Example

In my previous role, I was responsible for adjusting the replication factor of several critical Kafka topics used by the company to enhance system fault tolerance and data availability. By following the steps above, we successfully increased the replication factor of some high-traffic topics from 1 to 3, significantly improving the stability and reliability of the messaging system.

Summary

In summary, changing the replication factor of a Kafka topic is a process that requires careful planning and execution. Proper operation ensures data security and high service availability.