Component Tuning Levers
Also available as:

Component Tuning Levers

There are a number of services that you can use to tune the performance of your Metron cluster. These services include Kafka, Storm, and HDFS. Within these services, you can modify parsers, enrichment, and indexing (Elasticsearch or Solr).

When you consider tuning your HCP architecture, it is important to note where you can modify settings. For example, Storm gives you the ability to independently set tasks in executors for parser topologies. This is important if you want to set the number of tasks higher than the number of executors to accommodate for future performance tuning and rebalancing without the need to bring down your topologies. However, for enrichment and indexing topologies, HCP uses Flux, and there is no method for specifying the number of tasks from the number of executors in Flux. By default, the number of tasks equals the number of executors.

The following lists the major properties for each service that you can modify to tune your cluster:

  • Kafka

    • Number partitions

  • Storm

    • Kafka spout

      • Polling frequency

      • Polling timeouts

      • Offset commit period

      • Max uncommitted offsets

    • Number workers (OS processes)

    • Number executors (threads in a process)

    • Number ackers

    • Max spout pending

    • Spout and bolt parallelism

  • HDFS

    • Replication factor

  • Indexing
    • Elasticsearch
    • Solr