How to move elasticsearch data from one server to another

When migrating Elasticsearch data from one server to another, several methods can be employed. Below are several commonly used methods:

1. Snapshot and Restore (Snapshot and Restore)

This is the officially recommended method for migrating data in Elasticsearch. Steps are as follows:

Step 1: Create a Snapshot Repository

First, configure a snapshot repository on the source server. This can be a filesystem repository or a supported cloud storage service.

json
PUT /_snapshot/my_backup
{
  "type": "fs",
  "settings": {
    "location": "/mount/backups/my_backup",
    "compress": true
  }
}

Step 2: Create a Snapshot

Then, create a snapshot on the source server that includes all indices to be migrated.

json
PUT /_snapshot/my_backup/snapshot_1
{
  "indices": "index_1,index_2",
  "ignore_unavailable": true,
  "include_global_state": false
}

Step 3: Configure the Same Snapshot Repository on the Destination Server

Ensure the destination server has access to the snapshot storage location and is configured with the same repository.

Step 4: Restore Data from the Snapshot

Finally, restore the snapshot on the destination server.

json
POST /_snapshot/my_backup/snapshot_1/_restore
{
  "indices": "index_1,index_2",
  "include_global_state": false
}

2. Using Elasticsearch-dump Tool

Elasticsearch-dump is a third-party tool used for exporting and importing data. It can handle large-scale data migrations.

Step 1: Install the Tool

bash
npm install elasticdump -g

Step 2: Export Data

bash
elasticdump --input=http://source-server:9200/my_index --output=my_index_mapping.json --type=mapping
elasticdump --input=http://source-server:9200/my_index --output=my_index_data.json --type=data

Step 3: Import Data

bash
elasticdump --input=my_index_mapping.json --output=http://destination-server:9200/my_index --type=mapping
elasticdump --input=my_index_data.json --output=http://destination-server:9200/my_index --type=data

3. Reindex from Remote

If both Elasticsearch clusters can communicate with each other, you can use the reindex from remote feature to migrate data directly from one cluster to another.

Step 1: Configure Remote Cluster on the Destination Cluster

First, configure reindex.remote.whitelist on the destination Elasticsearch cluster to allow reading from the source cluster.

Step 2: Use _reindex to Migrate Data

json
POST /_reindex
{
  "source": {
    "remote": {
      "host": "http://source-server:9200",
      "username": "user",
      "password": "pass"
    },
    "index": "source_index"
  },
  "dest": {
    "index": "destination_index"
  }
}

When using any of the above methods, ensure data consistency and integrity while also prioritizing security, particularly encryption and access control during data transmission. Each method has specific use cases, and the choice depends on business requirements and environment configurations.