Administration
Also available as:
PDF
loading table of contents...

Indexing

The Indexing topology takes data ingested into Kafka from enriched topologies and sends the data to an indexing bolt configured to write to one or more of the following indices:

  • Elasticsearch or Solr

  • HDFS under /apps/metron/enrichment/indexed

Indices are written in batch and the batch size is specified in the Enrichment Configuration file by the batchSize parameter. This configuration is variable by sensor type.

Errors during indexing are sent to a Kafka topic named indexing_error.

The following figure illustrates the data flow between Kafka, the Indexing topology, and HDFS.

Figure 5.2. Indexing Architecture