Appendix A. Property Mappings
This section contains mappings for the various performance tuning properties for parsers,enrichment, and indexing and how they are materialized.
Parsers
Table A.1.
| Category | Management UI Property Name | JSON Config File Property Name | CLI Option | Storm Property Name | Notes |
|---|---|---|---|---|---|
| Storm topology config | Num Workers | n/a | -nw,--num_workers <NUM_WORKERS> | topology.workers | |
| Num Ackers | n/a | --na,--num_ackers <NUM_ACKERS> | topology.acker-executors | ||
| Storm Config | topology.max-spout.pending | -e,--extra_topology_options <JSON_FILE>, e.g., {
"topology.max.spout.pending" : NUM } | topology.max.spout.pending | Put property in JSON format in a file named
storm-<MY_PARSER>-config.json | |
| Kafka | Spout Parallelism | n/a | -sp,--spout_p <SPOUT_PARALLELISM_HINT> | n/a | |
| Spout Num Tasks | n/a | -snt,--spout_num_tasks <NUM_TASKS> | n/a | ||
| Spout Config | spout.pollTimeoutMs | -esc,--extra_kafka_spout_config <JSON_FILE>, e.g. { "spout.pollTimeoutMs"
: 200 } | n/a | Put property in JSON format in a file named
spout-<MY_PARSER>-config.json | |
| Spout Config | spout.maxUncommittedOffsets | -esc,--extra_kafka_spout_config <JSON_FILE>, e.g. {
"spout.maxUncommittedOffsets" : 10000000 } | n/a | Put property in JSON format in a file named
spout-<MY_PARSER>-config.json | |
| Spout Config | spout.offsetCommitPeriodMs | -esc,--extra_kafka_spout_config <JSON_FILE>, e.g. {
"spout.offsetCommitPeriodMs" : 30000 } | n/a | Put property in JSON format in a file named
spout-<MY_PARSER>-config.json | |
| Parser bolt | Parser Num Tasks | n/a | -pnt,--parser_num_tasks <NUM_TASKS> | n/a | |
| Parser Parallelism | n/a | -pp,--parser_p <PARALLELISM_HINT> | n/a | ||
| Parser Parallelism | n/a | -pp,--parser_p <PARALLELISM_HINT> | n/a |
Enrichments
Table A.2.
| Category | Ambari Property Name | enrichment-splitjoin.properties Property | Flux Property | Flux Section Location | Storm Property Name | Notes |
|---|---|---|---|---|---|---|
| Storm topology config | enrichment_workers | enrichment.workers | topology.workers | line 18, config | topology.workers | |
| enrichment_acker_executors | enrichment.acker_executors | topology.ackers.executors | line 18, config | topology.ackers.executors | ||
| enrichment_topology_max_spout_pending | topology.max.spout.pending | topology.max.spout.pending | line 18, config | topology.max.spout.pending | ||
| Kafka spout | enrichment_kafka_spout_parallelism | kafka.spout.parallelism | parallelism | line 245, id: kafkaSpout | n/a | |
| n/a | n/a | session.timeout.ms | line 201, id: kafkaProps | n/a | Kafka consumer client property | |
| n/a | n/a | enable.auto.commit | line 201, id: kafkaProps | n/a | Kafka consumer client property | |
| n/a | n/a | setPollTimeoutMs | line 230, id: kafkaConfig | n/a | Kafka consumer client property | |
| n/a | n/a | setMaxUncommittedOffsets | line 230, id: kafkaConfig | n/a | Kafka consumer client property | |
| n/a | n/a | setOffsetCommitPeriodMs | line 230, id: kafkaConfig | n/a | Kafka consumer client property | |
| Enrichment splitter | enrichment_split_parallelism | enrichment_split_parallelism | parallelism | line 253, id: enrichmentSplitBolt | n/a | |
| Enrichment joiner | enrichment_join_parallelism | enrichment_join_parallelism | parallelism | line 316, id: enrichmentJoinBolt | n/a | |
| Threat intel splitter | threat_intel_split_parallelism | threat_intel_split_parallelism | parallelism | line 338, id: threatIntelSplitBolt | n/a | |
| Threat intel joiner | threat_intel_join_parallelism | threat_intel_join_parallelism | parallelism | line 376, id: threatIntelJoinBolt | n/a | |
| Output bolt | kafka_writer_parallelism | kafka_writer_parallelism | parallelism | line 397, id: | n/a |
Indexing
| Category | Ambari Property Name | hdfs.properties property | Flux Property | Flux Section Location | Storm Property Name | Notes |
|---|---|---|---|---|---|---|
| Storm topology config | enrichment_workers | enrichment.workers | topology.workers | line 19, config | topology.workers | |
| enrichment_acker_executors | enrichment.acker.executors | topology.acker.executors | line 19, config | topology.acker.executors | ||
| enrichment_topology_max_spout_pending | topology.max.spout.pending | topology.max.spout.pending | line 19, config | topology.max.spout.pending | ||
| Kafka spout | batch_indexing_kafka_spout_parallelism | kafka.spout.parallelism | parallelism | line 123, id: kafkaSpout | n/a | |
| n/a | session.timeout.ms | session.timeout.ms | line 80, id: kafkaProps | n/a | Kafka consumer client property | |
| n/a | enable.auto.commit | enable.auto.commit | line 80, id: kafkaProps | n/a | Kafka consumer client property | |
| n/a | n/a | setPollTimeoutMs | line 108, id: kafkaConfig | n/a | Kafka consumer client property | |
| n/a | n/a | setMaxUncommittedOffsets | line 108, id: kafkaConfig | n/a | Kafka consumer client property | |
| n/a | n/a | setOffsetCommitPeriodMs | line 108, id: kafkaConfig | n/a | Kafka consumer client property | |
| Output bolt | hdfs_writer_parallelism | hdfs.writer.parallelism | parallelism | line 133, id: hdfsIndexingBolt | n/a | |
| n/a | n/a | hdfsSyncPolicy | line 47, id: hdfsWriter | n/a | See notes below about adding this prop | |
| bolt_hdfs_rotation_policy_units | bolt.hdfs.rotation.policy.units | constructorArgs | line 41, id: hdfsRotationPolicy | n/a | ||
| bolt_hdfs_rotation_policy_count | bolt.hdfs.rotation.policy.count | constructorArgs | line 41, id: hdfsRotationPolicy | n/a |