Use Case Specific Tuning Suggestions
Also available as:
PDF

View Kafka Offset Lags Example

You can use the Kafka consumer group offset lag viewer to monitor the delta calculations between the current and end offset for a partition.

  1. Set up some environment variables.
    export BROKERLIST  your broker comma-delimated list of host:ports>
    export ZOOKEEPER  your zookeeper comma-delimated list of host:ports>
    export KAFKA_HOME  kafka home dir>
    export METRON_HOME  your metron home>
    export HDP_HOME  your HDP home>
    
  2. If you have Kerberos enabled, set up the security protocol.
    $ cat /tmp/consumergroup.config
    security.protocol=SASL_PLAINTEXT
    
  3. Enter the following command to display a table containing offsets for all partitions and consumers associated with a running topology's consumer group:
    ${KAFKA_HOME}/bin/kafka-consumer-groups.sh \ --command-config=/tmp/consumergroup.config \ --describe \ --group enrichments \ --bootstrap-server $BROKERLIST \ --new-consumer
    The command displays the following table:
    GROUP                          TOPIC              PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             OWNER
    enrichments                    enrichments        9          29746066        29746067        1               consumer-2_/xxx.xxx.xxx.xxx
    enrichments                    enrichments        3          29754325        29754326        1               consumer-1_/xxx.xxx.xxx.xxx
    enrichments                    enrichments        43         29754331        29754332        1               consumer-6_/xxx.xxx.xxx.xxx
    ...
    
    Note
    Note
    Output displays only when the topology is running because the consumer groups only exist while consumers in the spouts are up and running.
    The LAG column lists the current delta calculation between the current and end offset for the partition. The column value indicates how close you are to keeping up with incoming data. It also indicates whether there are any problems with specific consumers getting stuck.
  4. To watch the offsets and lags change over time, add a watch command and set the refresh rate to 10 seconds:
    watch -n 10 -d ${KAFKA_HOME}/bin/kafka-consumer-groups.sh \
        --command-config=/tmp/consumergroup.config \
        --describe \
        --group enrichments \
        --bootstrap-server $BROKERLIST \
        --new-consumer
    
    The watch command runs every 10 seconds and refreshes the screen with new information. The command also highlights the differences from the current output and the last output screens.