Run Book - Technical Preview
Also available as:
PDF

Chapter 7. Configuring Indexing

The indexing topology is a topology dedicated to taking the data from a topology that has been enriched and storing the data in one or more supported indices. More specifically, the enriched data is ingested into Kafka, written in an indexing batch or bolt with a specified size, and sent to one or more specified indices. The configuration is intended to configure the indexing used for a given sensor type (for example, snort).

Currently, HCP supports the following indices:

  • Elasticsearch

  • Solr

  • HDFS under /apps/metron/indexing/indexed

Depending on how you start the indexing topology, it can have HDFS and either elasticsearch or SOLR writers running.

Just like the Global Configuration file, the Indexing Configuration file format is a JSON file stored in ZooKeeper and on disk at $METRON_HOME/config/zookeeper/ indexing.

[Note]Note

Errors during indexing are sent to a Kafka queue called index_errors

Within the sensor-specific configuration, you can configure the individual writers. The parameters currently supported are:

index

The name of the index to write to (defaulted to the name of the sensor).

batchSize

The size of the batch that is written to the indices at once (defaulted to 1).

enabled

Whether the index or writer is enabled (default true).