Tuning Guide
Also available as:
PDF

Enrichment Tuning Example

We landed on the same number of partitions for enrichment and indexing as we did for bro - 48.

For configuring Storm, there is a flux file and properties file that we modified. Here are the settings we changed for Bro in Flux. +Note that the main Metron-specific option we've changed to accommodate the desired rate of data throughput is max cache size in the join bolts.

More information on Flux can be found here - https://storm.apache.org/releases/1.1.0/flux.html

general storm settings

 topology.workers: 8
 topology.acker.executors: 48
 topology.max.spout.pending: 2000
 

Spout and Bolt Settings

 kafkaSpout
     parallelism=48
     session.timeout.ms=29999
     enable.auto.commit=false
     setPollTimeoutMs=200
     setMaxUncommittedOffsets=10000000
     setOffsetCommitPeriodMs=30000
 enrichmentSplitBolt
     parallelism=4
 enrichmentJoinBolt
     parallelism=8
     withMaxCacheSize=200000
     withMaxTimeRetain=10
 threatIntelSplitBolt
     parallelism=4
 threatIntelJoinBolt
     parallelism=4
     withMaxCacheSize=200000
     withMaxTimeRetain=10
 outputBolt
     parallelism=48