Tuning Guide
Also available as:
PDF

Modifying Index Parameters Using Flux (Advanced)

Some of the tuning indexing properties, for example the HDFS sync policy setting, can only be modified using Flux. However, if you manually change your Flux file, if you perform an upgrade, Ambari will overwrite all of your changes. Be sure to back up your Flux changes prior to performing an upgrade.

[Important]Important

You should be familiar with Storm Flux before you adjust the values in this section. Changes to Flux file properties that are managed by Ambari will render Ambari unable to further manage the property.

You can find the indexing Flux file at $METRON_HOME/flux/indexing/batch/remote.yaml.

CategoryFlux PropertyFlux Section LocationSuggested Value
Kafka spoutsession.timeout.msline 80, id: kafkaPropsKafka consumer client property
 enable.auto.commitline 80, id: kafkaPropsKafka consumer client property
 setPollTimeoutMsline 108, id: kafkaConfigKafka consumer client property
 setMaxUncommittedOffsetsline 108, id: kafkaConfigKafka consumer client property
 setOffsetCommitPeriodMsline 108, id: kafkaConfigKafka consumer client property
Output bolthdfsSyncPolicyline 47, id: hdfsWriterSee notes below about adding this prop

To modify index tuning properties, complete the following steps:

  1. Add a new setter to the hdfsWriter around line 56.

     53             -   name: "withRotationPolicy"
     54                 args:
     55                     - ref: "hdfsRotationPolicy
     56             -   name: "withSyncPolicy"
     57                 args:
     58                     - ref: "hdfsSyncPolicy
    

    Lines are 53-55 provided for context.

  2. Add an hdfsSyncPolicy after the hdfsRotationPolicy that appears on line 41:

     41     -   id: "hdfsRotationPolicy"
    ...
     45           - "${bolt.hdfs.rotation.policy.units}"
     46
     47     -   id: "hdfsSyncPolicy"
     48         className: "org.apache.storm.hdfs.bolt.sync.CountSyncPolicy"
     49         constructorArgs:
     50           -  100000