Component Tuning Variables Overview

There are a number of services that you can use to tune the performance of your Metron cluster. These services include Kafka, Storm, and HDFS. Within these services, you can modify variables for topologies, including parsers, enrichment, and indexing (Elasticsearch or Solr).

When you consider tuning your HCP architecture, it is important to note where you can modify variables. For example, Storm gives you the ability to independently set tasks in executors for parser topologies. This is important if you want to set the number of tasks higher than the number of executors to accommodate for future performance tuning and rebalancing without the need to bring down your topologies. However, for enrichment and indexing topologies, HCP uses Flux, and there is no method for specifying the number of tasks from the number of executors in Flux. By default, the number of tasks equals the number of executors.

The following lists the major variables for each service that you can modify to tune your cluster:

• Kafka

  • Number partitions

• Storm

  • Kafka spout

    • Polling frequency

    • Polling timeouts

    • Offset commit period

    • Max uncommitted offsets

  • Number workers (OS processes)

  • Number executors (threads in a process)

  • Number ackers

  • Max spout pending

  • Spout and bolt parallelism

• HDFS

  • Batch size

  • Replication factor

• Indexing
  • Elasticsearch
  • Solr