Tuning Guide
Also available as:


You can use the following Storm and Kafka tools to monitor your topologies.


  • Consumer group offset lag viewer

  • Kafka Manager - A GUI tool to facilitate creating, modifying, and managing your Kafka topics

  • Console consumer - useful for quickly verifying topic contents


The Storm user interface is a very useful tool for monitoring your topologies. For more information on the Storm user interface, see Reading and Understanding the Storm UI.

Example: Viewing Kafka Offset Lags

You can use the Kafka consumer group offset lag viewer to monitor the delta calculations between the current and end offset for a partition.

  1. Set up some environment variables.

    export BROKERLIST  your broker comma-delimated list of host:ports>
    export ZOOKEEPER  your zookeeper comma-delimated list of host:ports>
    export KAFKA_HOME  kafka home dir>
    export METRON_HOME  your metron home>
    export HDP_HOME  your HDP home>
  2. If you have Kerberos enabled, set up the security protocol.

    $ cat /tmp/consumergroup.config
  3. Enter the following command to display a table containing offsets for all partitions and consumers associated with a running topology's consumer group:

    ${KAFKA_HOME}/bin/kafka-consumer-groups.sh \
        --command-config=/tmp/consumergroup.config \
        --describe \
        --group enrichments \
        --bootstrap-server $BROKERLIST \

    The command displays the following table:

    GROUP                          TOPIC              PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             OWNER
    enrichments                    enrichments        9          29746066        29746067        1               consumer-2_/xxx.xxx.xxx.xxx
    enrichments                    enrichments        3          29754325        29754326        1               consumer-1_/xxx.xxx.xxx.xxx
    enrichments                    enrichments        43         29754331        29754332        1               consumer-6_/xxx.xxx.xxx.xxx

    Output displays only when the topology is running because the consumer groups only exist while consumers in the spouts are up and running.

    The LAG column lists the current delta calculation between the current and end offset for the partition. The column value indicates how close you are to keeping up with incoming data. It also indicates whether there are any problems with specific consumers getting stuck.

  4. To watch the offsets and lags change over time, add a watch command and set the refresh rate to 10 seconds:

    watch -n 10 -d ${KAFKA_HOME}/bin/kafka-consumer-groups.sh \
        --command-config=/tmp/consumergroup.config \
        --describe \
        --group enrichments \
        --bootstrap-server $BROKERLIST \

    The watch command runs every 10 seconds and refreshes the screen with new information. The command also highlights the differences from the current output and the last output screens.