Elasticsearch Reindex

Elasticsearch로 AP의 logging을 하다보면 서버의 메모리 용량이 충분하더라도 (서버 memory 64GB 중 ES에 30GB 할당)

shard의 개수가 12000개 이상이 되면 indexing 혹은 process가 중지되어 버리는 문제를 경험했다.

shard를 줄이기 위하여 더이상 쓰기가 필요하지 않은 index를 하나로 묶는 방법을 택했다.

 

Elasticsearch Reindex API
To reindex data in Elasticsearch, you can use the Reindex API. The Reindex API allows you to copy documents from one index to another, making it useful for tasks like changing mappings, updating documents, or creating new indices.

Here's an example of how to use the Reindex API:

Create a new index to hold the reindexed data. This step is optional if you want to reindex data within the same index.

POST /new_index
{
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 0
  }
}

Use the Reindex API to copy documents from the source index to the destination index.

POST _reindex
{
  "source": {
    "index": "source_index"
  },
  "dest": {
    "index": "new_index"
  }
}

Replace source_index with the name of the index you want to reindex from and new_index with the name of the index you created or the index you want to reindex into. You can also specify additional parameters like query filters to select specific documents for reindexing.

Monitor the reindexing process. Depending on the size of your data, this operation may take some time to complete. You can track the progress by monitoring the response of the reindexing request or by using the Task API.

Once the reindexing is complete, you can perform any necessary cleanup or additional steps as required.

It's important to note that reindexing can have performance implications, especially on large datasets. It's recommended to perform reindexing operations during low-traffic periods or when the Elasticsearch cluster is not under heavy load.

Additionally, remember to adjust any existing applications or queries to use the new index name if you've created a new index for the reindexed data.