Hortonworks Cybersecurity Platform
Also available as:
PDF
loading table of contents...

Prune Data from Elasticsearch

Elasticsearch provides tooling to prune index data through its Curator utility.

  1. Use the following command to prune the Elasticsearch data:
    The following is a sample invocation that you can configure through Cron to prune indexes based on the timestamp in the index name.
    /opt/elasticsearch-curator/curator_cli --host localhost delete_indices --filter_list '
         {
           "filtertype": "age",
           "source": "name",
           "timestring": "%Y.%m.%d",
           "unit": "days",
           "unit_count": 10,
           "direction": "older”
         }'
    Using name as the source value causes Curator to look for a timestring value within the index or snapshot name, and to convert that into an epoch timestamp (epoch implies UTC).
  2. For finer-grained control over indexes pruning, provide multiple filters as an array of JSON objects to filter_list. Chaining multiple filters implies logical AND.
    --filter_list '[{"filtertype":"age","source":"creation_date","direction":"older","unit":"days","unit_count":13},
    {"filtertype":"pattern","kind":"prefix","value":"logstash"}]'
    For finer-grained control over the indexes pruning that will be pruned, you can also provide multiple filters as an array of JSON objects to filter_list. Chaining multiple filters implies there is an implicit logical AND when chaining multiple filters.
    --filter_list '[{"filtertype":"age","source":"creation_date","direction":"older","unit":"days","unit_count":13},
    {"filtertype":"pattern","kind":"prefix","value":"logstash"}]'