Tuning Topologies
Also available as:
PDF

Tune Additional Enrichment Storm Settings

After the number of enrichment executors has been determined and thoroughly tested, you can set or modify the last remaining Storm parameters.

  1. Based on the capacity you've seen during testing, reduce the overall number of ackers.
    Alternatively, you an leave a single acker per worker as it will ensure that there are no messages sent between Storm workers over the network interface.
  2. Set the Max Spout Pending parameter such that the maximum number of unacked tuples in the topology is close to the Parser Executor capacity (for example, ~0.950).
    If this is the case then it can be assured that if there is a large spike in incoming events, the topology will not become overloaded. An example approach to determine this value would be to increase the producer events per second by a large amount and test various values for Max Spout Pending. The value can be set under the Storm settings of the relevant Parser.
    vi ~/enrichment.properties
    ##### Storm #####
    enrichment.workers=3
    enrichment.acker.executors=3
    topology.worker.childopts=
    topology.auto-credentials=[]
    topology.max-spout.pending=
  3. Check the Executor capacity.
    The executor capacity should not exceed ~0.950. Assuming the number of events generated by the producer is far greater than the capacity of the Parser topology, capacity is the only value that needs to be monitored in the Storm UI.
  4. If you need to increase the Error Writer Num Executors value, you can directly modify the Flux file and include the "parallelism" parameter under the appropriate Storm Bolt declarations.
    sudo vi /usr/hcp/current/metron/flux/enrichment/remote-unified.yaml
    id: "enrichmentErrorOutputBolt"
    className: "org.apache.metron.writer.bolt.BulkMesageWriterBolt"
    constructorArgs:
         - "${kafka.zk}"
    configMethods:
         - name: "withMessageWriter"
                   args:
                        - ref: "erichmentErrorKafkaWriter"
    parallelism: 3
    Generally, since a small number of errors is expected, the Error Writer Num Executors value does not need to be increased.