Enrichment Tuning Example
We landed on the same number of partitions for enrichment and indexing as we did for bro - 48.
For configuring Storm, there is a flux file and properties file that we modified. Here are the settings we changed for Bro in Flux. +Note that the main Metron-specific option we've changed to accommodate the desired rate of data throughput is max cache size in the join bolts.
More information on Flux can be found here - https://storm.apache.org/releases/1.1.0/flux.html
general storm settings
topology.workers: 8 topology.acker.executors: 48 topology.max.spout.pending: 2000
Spout and Bolt Settings
kafkaSpout parallelism=48 session.timeout.ms=29999 enable.auto.commit=false setPollTimeoutMs=200 setMaxUncommittedOffsets=10000000 setOffsetCommitPeriodMs=30000 enrichmentSplitBolt parallelism=4 enrichmentJoinBolt parallelism=8 withMaxCacheSize=200000 withMaxTimeRetain=10 threatIntelSplitBolt parallelism=4 threatIntelJoinBolt parallelism=4 withMaxCacheSize=200000 withMaxTimeRetain=10 outputBolt parallelism=48