thecase/elasticsearch-curator repository overview

⁠Elasticsearch Curator

A container to be run once daily that will perform the following:

• snapshot (backup) Kibana
• snapshot logstash-* and syslog-* indices before today
• tune down replica count to one (1) on all indices that have been snapshotted
• close indices older than 14 days
• delete indices older than 30 days
• delete snapshots older than 365 days

⁠Prerequisites

⁠Azure Storage credentials in the Elasticsearch Keystore

The elk-data stack's keystore service will take care of applying the Azure Storage account data to the Elasticsearch cluster.

If you want to verify that the azure.client.default keystore items have already been set, you can get shell on either an Elasticsearch master or data node and issue the following command:

bin/elasticsearch-keystore list

The Azure keystore items need to exist on every Elasticsearch master and data node in the cluster.

If any node is missing the keystore and you need to set it up manually, you will have to open a shell on each Elasticsearch node. Then you can enter the Azure information and restart that container (IMPORTANT be mindful of shard allocation and cluster state when restarting data nodes!). In this example, elksnapshots should be replaced with your storage endpoint, without the protocol or .blob.microsoft.net tld.

echo elksnapshots | bin/elasticsearch-keystore add --stdin azure.client.default.account
bin/elasticsearch-keystore add azure.client.default.key

The last command will prompt you for a silent entry of the secret key.

⁠Setting up the Azure Storage repository

You need to have a repository set up before snapshots can be created. You can check to see if this repository information has already been applied with the following REST command in Cerebro:

method: GET
command: _snapshot 

If there is no repository listed, you'll need to create it. Please make sure the elasticsearch-snapshots container is created in the Blob section of the Azure Storage Account. Again, using Cerebro, enter the following in the REST tool:

method: PUT
command: _snapshot/azure-storage
data:
{
    "type": "azure",
    "settings": {
        "container": "elasticsearch-snapshots",
        "base_path": "/",
        "chunk_size": "64m",
        "compress": true
    }
}

⁠Running elasticsearch-curator

Note the cron schedule. It should start at 1am UTC, to give the prior daily index time to finalize.

Add to docker-compose.yml

curator:
  image: sonatype-docker-internal.steelcase.com/sctechdev/elasticsearch-curator:latest
  links:
    - client:elasticsearch
  labels:
    cron.action: "start"                                      
    cron.schedule: "0 1 * * *"                                
    io.rancher.container.start_once: true           

⁠Logging

Logs are sent to stdout in default format. Please create the proper Logstash filters to parse the messages for alerting purposes in the case of a failure.

⁠Restoring indices

note that the selected indices to be restore must either be closed, or not exist in the destination cluster, or the restore will stop with an error.

POST _snapshot/azure-storage/curator-2018.03.07/_restore
{
  "indices": "logstash-2017.12.17",
  "index_settings": {
    "index.number_of_replicas": 1
  },
  "ignore_index_settings": [
    "index.refresh_interval"
  ],
  "ignore_unavailable": true,
  "include_global_state": true
}